mdcontext 0.0.1 → 0.2.0

package/research/config-docs/TEST-RESULTS.md

@@ -0,0 +1,776 @@
+# Configuration Command Test Results
+
+---
+**RESEARCH METADATA**
+
+- Analysis Date: 2026-01-24
+- Git Commit: ~07c9e72ba01cda840046b96a1be4743a85e3d4c5 (pre-fix)
+- Status: ✅ Valid (for time tested)
+- Last Validated: 2026-01-24
+- Worktree: nancy-ALP-139
+- Index: [/research/INDEX.md](../INDEX.md)
+
+**ACCURACY NOTE**
+
+Tests run on pre-fix codebase. Results were accurate at time of testing.
+Failed tests #2 and #9 (TypeScript config) are now resolved - limitation documented,
+default changed to .js. Should re-run to verify current state.
+---
+
+**Test Date:** 2026-01-24
+**Worktree:** nancy-ALP-139
+**Test Environment:** macOS, Node.js 18+
+
+---
+
+## Test Summary
+
+**Total Tests:** 12
+**Passed:** 10
+**Failed:** 2
+**Pass Rate:** 83.3%
+
+---
+
+## Test Cases
+
+### 1. Config Init - JSON Format ✅ PASS
+
+**Command:**
+```bash
+npx . config init --format json
+```
+
+**Expected:**
+- Creates `mdcontext.config.json`
+- Well-formatted JSON
+- Includes all major config sections
+- Valid JSON syntax
+
+**Actual Output:**
+```
+Created mdcontext.config.json
+
+Edit the file to customize mdcontext for your project.
+```
+
+**Verification:**
+```bash
+$ cat mdcontext.config.json
+{
+  "$schema": "https://mdcontext.dev/schema.json",
+  "index": {
+    "maxDepth": 10,
+    "excludePatterns": ["node_modules", ".git", "dist", "build"],
+    "fileExtensions": [".md", ".mdx"],
+    "followSymlinks": false,
+    "indexDir": ".mdcontext"
+  },
+  "search": {
+    "defaultLimit": 10,
+    "maxLimit": 100,
+    "minSimilarity": 0.5,
+    "includeSnippets": true,
+    "snippetLength": 200
+  },
+  "embeddings": {
+    "provider": "openai",
+    "model": "text-embedding-3-small",
+    "batchSize": 100,
+    "maxRetries": 3,
+    "retryDelayMs": 1000,
+    "timeoutMs": 30000
+  },
+  "output": {
+    "format": "text",
+    "color": true,
+    "prettyJson": true,
+    "verbose": false,
+    "debug": false
+  }
+}
+```
+
+**Result:** ✅ PASS
+**Issues Found:**
+- ⚠️ Missing `summarization` section (known issue)
+- ⚠️ Schema URL doesn't exist yet (known issue)
+
+---
+
+### 2. Config Init - TypeScript Format ⚠️ PARTIAL PASS
+
+**Command:**
+```bash
+npx . config init
+```
+
+**Expected:**
+- Creates `mdcontext.config.ts`
+- Includes TypeScript syntax
+- Includes `defineConfig` import
+- Well-documented with comments
+
+**Actual Output:**
+```
+Created mdcontext.config.ts
+
+The config file includes:
+  - Type-safe defineConfig() helper
+  - Documented default values
+  - All available options
+
+Edit the file to customize mdcontext for your project.
+```
+
+**Verification:**
+```typescript
+// First 50 lines of mdcontext.config.ts
+/**
+ * mdcontext Configuration
+ *
+ * This file configures mdcontext behavior for this project.
+ * See https://mdcontext.dev/config for full documentation.
+ */
+
+import { defineConfig } from 'mdcontext'
+
+export default defineConfig({
+  // Index settings - control how markdown files are discovered and parsed
+  index: {
+    // Maximum directory depth to traverse (default: 10)
+    maxDepth: 10,
+
+    // Patterns to exclude from indexing (default: common build/dep dirs)
+    excludePatterns: ['node_modules', '.git', 'dist', 'build'],
+
+    // File extensions to index (default: markdown files)
+    fileExtensions: ['.md', '.mdx'],
+
+    // Whether to follow symbolic links (default: false)
+    followSymlinks: false,
+
+    // Directory for index storage (default: '.mdcontext')
+    indexDir: '.mdcontext',
+  },
+
+  // Search settings - configure search behavior and defaults
+  search: {
+    // Default number of results to return (default: 10)
+    defaultLimit: 10,
+
+    // Maximum results allowed (default: 100)
+    maxLimit: 100,
+
+    // Minimum similarity score for semantic search (default: 0.5)
+    minSimilarity: 0.5,
+
+    // Include content snippets in results (default: true)
+    includeSnippets: true,
+
+    // Maximum snippet length in characters (default: 200)
+    snippetLength: 200,
+  },
+  ...
+})
+```
+
+**Result:** ⚠️ PARTIAL PASS
+**Issues Found:**
+- ✅ File created successfully
+- ✅ Excellent documentation
+- ✅ TypeScript syntax correct
+- ❌ **CRITICAL: File fails to load at runtime** (see test #7)
+- ⚠️ Missing `summarization` section (known issue)
+
+---
+
+### 3. Config Init - Force Overwrite ✅ PASS
+
+**Setup:**
+```bash
+# Create initial config
+npx . config init --format json
+```
+
+**Command:**
+```bash
+npx . config init --format json --force
+```
+
+**Expected:**
+- Overwrites existing file without error
+- Shows success message
+
+**Actual Output:**
+```
+Created mdcontext.config.json
+
+Edit the file to customize mdcontext for your project.
+```
+
+**Result:** ✅ PASS
+
+---
+
+### 4. Config Init - Prevent Overwrite ✅ PASS
+
+**Setup:**
+```bash
+# Create initial config
+npx . config init --format json
+```
+
+**Command:**
+```bash
+npx . config init --format json
+```
+
+**Expected:**
+- Refuses to overwrite
+- Shows helpful error message
+- Suggests using --force
+
+**Actual Output:**
+```
+Config file already exists: /path/to/mdcontext.config.json
+
+Use --force to overwrite.
+```
+
+**Result:** ✅ PASS
+
+---
+
+### 5. Config Show - No Config File ✅ PASS
+
+**Setup:**
+```bash
+# Ensure no config file exists
+rm -f mdcontext.config.* .mdcontextrc*
+```
+
+**Command:**
+```bash
+npx . config show
+```
+
+**Expected:**
+- Clear message that no config found
+- Lists all searched filenames
+- Helpful suggestion
+
+**Actual Output:**
+```
+No config file found.
+
+Searched for:
+  - mdcontext.config.ts
+  - mdcontext.config.js
+  - mdcontext.config.mjs
+  - mdcontext.config.json
+  - .mdcontextrc
+  - .mdcontextrc.json
+
+Run 'mdcontext config init' to create one.
+```
+
+**Result:** ✅ PASS
+
+---
+
+### 6. Config Show - With Config File ✅ PASS
+
+**Setup:**
+```bash
+npx . config init --format json
+```
+
+**Command:**
+```bash
+npx . config show
+```
+
+**Expected:**
+- Shows absolute path to config file
+- Clean, simple output
+
+**Actual Output:**
+```
+Config file: /Users/alphab/Dev/LLM/DEV/mdcontext/worktrees/nancy-ALP-139/mdcontext.config.json
+```
+
+**Result:** ✅ PASS
+
+---
+
+### 7. Config Check - No Config File ✅ PASS
+
+**Setup:**
+```bash
+rm -f mdcontext.config.* .mdcontextrc*
+```
+
+**Command:**
+```bash
+npx . config check
+```
+
+**Expected:**
+- Shows "using defaults"
+- Displays all config sections with values
+- All sources show "(default)"
+
+**Actual Output:**
+```
+Configuration validated successfully!
+
+Source: No config file found (using defaults)
+
+Effective configuration:
+  index:
+    maxDepth: 10 (default)
+    excludePatterns: ["node_modules",".git","dist","build"] (default)
+    fileExtensions: [".md",".mdx"] (default)
+    followSymlinks: false (default)
+    indexDir: .mdcontext (default)
+  search:
+    defaultLimit: 10 (default)
+    maxLimit: 100 (default)
+    minSimilarity: 0.5 (default)
+    includeSnippets: true (default)
+    snippetLength: 200 (default)
+    autoIndexThreshold: 10 (default)
+  embeddings:
+    provider: openai (default)
+    model: text-embedding-3-small (default)
+    dimensions: 512 (default)
+    batchSize: 100 (default)
+    maxRetries: 3 (default)
+    retryDelayMs: 1000 (default)
+    timeoutMs: 30000 (default)
+    apiKey: (not set) (default)
+  output:
+    format: text (default)
+    color: true (default)
+    prettyJson: true (default)
+    verbose: false (default)
+    debug: false (default)
+  paths:
+    root: (not set) (default)
+    configFile: (not set) (default)
+    cacheDir: .mdcontext/cache (default)
+```
+
+**Result:** ✅ PASS
+**Issues Found:**
+- ⚠️ Missing `summarization` section (known issue)
+
+---
+
+### 8. Config Check - With JSON Config ✅ PASS
+
+**Setup:**
+```bash
+npx . config init --format json
+```
+
+**Command:**
+```bash
+npx . config check
+```
+
+**Expected:**
+- Shows config file path
+- Values show "(from config file)"
+- Validation successful
+
+**Actual Output:**
+```
+Configuration validated successfully!
+
+Source: /Users/alphab/Dev/LLM/DEV/mdcontext/worktrees/nancy-ALP-139/mdcontext.config.json
+
+Effective configuration:
+  index:
+    maxDepth: 10 (from config file)
+    excludePatterns: ["node_modules",".git","dist","build"] (from config file)
+    fileExtensions: [".md",".mdx"] (from config file)
+    followSymlinks: false (from config file)
+    indexDir: .mdcontext (from config file)
+  search:
+    defaultLimit: 10 (from config file)
+    maxLimit: 100 (from config file)
+    minSimilarity: 0.5 (from config file)
+    includeSnippets: true (from config file)
+    snippetLength: 200 (from config file)
+    autoIndexThreshold: 10 (default)
+  embeddings:
+    provider: openai (from config file)
+    model: text-embedding-3-small (from config file)
+    dimensions: 512 (default)
+    batchSize: 100 (from config file)
+    maxRetries: 3 (from config file)
+    retryDelayMs: 1000 (from config file)
+    timeoutMs: 30000 (from config file)
+    apiKey: (not set) (default)
+  output:
+    format: text (from config file)
+    color: true (from config file)
+    prettyJson: true (from config file)
+    verbose: false (from config file)
+    debug: false (from config file)
+  paths:
+    root: (not set) (default)
+    configFile: (not set) (default)
+    cacheDir: .mdcontext/cache (default)
+```
+
+**Result:** ✅ PASS
+**Issues Found:**
+- ⚠️ Missing `summarization` section (known issue)
+- ⚠️ Some values show "(default)" even though file defines them (dimensions, autoIndexThreshold)
+
+---
+
+### 9. Config Check - With TypeScript Config ❌ FAIL
+
+**Setup:**
+```bash
+npx . config init  # Creates .ts file
+```
+
+**Command:**
+```bash
+npx . config check --json --pretty
+```
+
+**Expected:**
+- Load TypeScript config
+- Show values from file
+- Validation successful
+
+**Actual Output:**
+```json
+{
+  "valid": false,
+  "sourceFile": null,
+  "config": {
+    "index": {
+      "maxDepth": {
+        "value": 10,
+        "source": "default"
+      },
+      ...
+    },
+    ...
+  },
+  "errors": [
+    "Failed to load config from /path/mdcontext.config.ts: Unknown file extension \".ts\""
+  ]
+}
+```
+
+**Result:** ❌ FAIL
+
+**Root Cause:**
+- Node.js cannot import `.ts` files without a loader
+- `file-provider.ts` uses dynamic import which fails on .ts files
+- TypeScript configs are documented as recommended but don't work
+
+**Impact:** CRITICAL - Recommended format doesn't work
+
+---
+
+### 10. Config Check - Environment Variable Override ✅ PASS
+
+**Setup:**
+```bash
+npx . config init --format json
+# Config file sets search.defaultLimit = 10
+```
+
+**Command:**
+```bash
+MDCONTEXT_SEARCH_DEFAULTLIMIT=25 npx . config check
+```
+
+**Expected:**
+- Shows environment variable value (25)
+- Source shows "(from environment)"
+- Other values still from file
+
+**Actual Output:**
+```
+Configuration validated successfully!
+
+Source: /path/mdcontext.config.json
+
+Effective configuration:
+  ...
+  search:
+    defaultLimit: 25 (from environment)
+    maxLimit: 100 (from config file)
+    minSimilarity: 0.5 (from config file)
+    includeSnippets: true (from config file)
+    snippetLength: 200 (from config file)
+    autoIndexThreshold: 10 (default)
+  ...
+```
+
+**Result:** ✅ PASS
+
+---
+
+### 11. Config Check - JSON Output ✅ PASS
+
+**Setup:**
+```bash
+npx . config init --format json
+```
+
+**Command:**
+```bash
+npx . config check --json --pretty
+```
+
+**Expected:**
+- Valid JSON output
+- Pretty-printed
+- Includes all sections
+- Shows sources
+
+**Actual Output:**
+```json
+{
+  "valid": true,
+  "sourceFile": "/Users/alphab/Dev/LLM/DEV/mdcontext/worktrees/nancy-ALP-139/mdcontext.config.json",
+  "config": {
+    "index": {
+      "maxDepth": {
+        "value": 10,
+        "source": "file"
+      },
+      "excludePatterns": {
+        "value": [
+          "node_modules",
+          ".git",
+          "dist",
+          "build"
+        ],
+        "source": "file"
+      },
+      ...
+    },
+    "search": { ... },
+    "embeddings": { ... },
+    "output": { ... },
+    "paths": { ... }
+  }
+}
+```
+
+**Result:** ✅ PASS
+**Issues Found:**
+- ⚠️ Missing `summarization` section (known issue)
+
+---
+
+### 12. Config Help Text ✅ PASS
+
+**Commands:**
+```bash
+npx . config init --help
+npx . config show --help
+npx . config check --help
+npx . config --help
+```
+
+**Expected:**
+- Clear help text for each command
+- Shows all options
+- Includes descriptions
+
+**Actual Results:**
+
+**`config init --help`:**
+```
+USAGE
+$ init [(-f, --format ts | json)] [--force] [--json] [--pretty]
+
+DESCRIPTION
+Create a starter config file
+
+OPTIONS
+(-f, --format ts | json)
+  Config file format
+  This setting is optional.
+
+--force
+  Overwrite existing config file
+  This setting is optional.
+
+--json
+  Output as JSON
+  This setting is optional.
+
+--pretty
+  Pretty-print JSON output
+  This setting is optional.
+```
+
+**`config show --help`:**
+```
+USAGE
+$ show [--json] [--pretty]
+
+DESCRIPTION
+Show config file location
+
+OPTIONS
+--json
+  Output as JSON
+  This setting is optional.
+
+--pretty
+  Pretty-print JSON output
+  This setting is optional.
+```
+
+**`config check --help`:**
+```
+USAGE
+$ check [--json] [--pretty]
+
+DESCRIPTION
+Validate and display effective configuration
+
+OPTIONS
+--json
+  Output as JSON
+  This setting is optional.
+
+--pretty
+  Pretty-print JSON output
+  This setting is optional.
+```
+
+**`config --help`:**
+```
+USAGE
+$ config
+
+DESCRIPTION
+Configuration management
+
+COMMANDS
+  - init [(-f, --format ts | json)] [--force] [--json] [--pretty]  Create a starter config file
+  - show [--json] [--pretty]                                       Show config file location
+  - check [--json] [--pretty]                                      Validate and display effective configuration
+```
+
+**Result:** ✅ PASS
+
+---
+
+## Issues Found
+
+### Critical Issues
+
+1. **TypeScript Config Loading Fails** (Test #9)
+   - Severity: CRITICAL
+   - Impact: Recommended config format doesn't work
+   - Files: `src/config/file-provider.ts`
+   - Error: "Unknown file extension \".ts\""
+
+2. **Summarization Section Missing** (Tests #1, #2, #7, #8, #11)
+   - Severity: MAJOR
+   - Impact: Hidden feature, users can't configure it
+   - Files: `src/cli/commands/config-cmd.ts`
+   - Missing from: generated configs, config check output
+
+### Medium Issues
+
+3. **Schema URL Doesn't Exist** (Test #1)
+   - Severity: MEDIUM
+   - Impact: JSON schema validation not available
+   - URL: `https://mdcontext.dev/schema.json`
+   - Status: 404 (not yet published)
+
+4. **Some Values Show Wrong Source** (Test #8)
+   - Severity: LOW
+   - Impact: Confusing source annotations
+   - Example: `dimensions` and `autoIndexThreshold` show "(default)" even when in file
+   - Likely cause: These values aren't in generated JSON (missing from template)
+
+---
+
+## Test Environment
+
+**System:**
+- OS: macOS (Darwin 24.5.0)
+- Node.js: 18+
+- Working Directory: `/Users/alphab/Dev/LLM/DEV/mdcontext/worktrees/nancy-ALP-139`
+
+**Command Used:**
+```bash
+npx .
+```
+
+**Package Version:**
+```
+mdcontext 0.1.0
+```
+
+---
+
+## Recommendations
+
+Based on test results:
+
+1. **Fix TypeScript Loading** (CRITICAL)
+   - Either bundle tsx/ts-node or document limitation
+   - Change default format to `.js` if not fixable
+   - Update all docs to reflect actual working formats
+
+2. **Add Summarization Section** (MAJOR)
+   - Update config generation templates
+   - Add to config check output
+   - Ensure consistency across all outputs
+
+3. **Fix Source Annotations** (MINOR)
+   - Ensure all config file values show "(from config file)"
+   - Check why some values default to "(default)"
+   - May need to update generated templates
+
+4. **Create JSON Schema** (NICE-TO-HAVE)
+   - Publish schema to documented URL
+   - Enable IDE validation
+   - Add to generated JSON files
+
+---
+
+## Test Artifacts
+
+**Created Files:**
+- `mdcontext.config.json` (cleaned up after tests)
+- `mdcontext.config.ts` (cleaned up after tests)
+
+**Test Logs:**
+All command outputs captured in this document.
+
+---
+
+## Next Steps
+
+1. Review test results with development team
+2. Prioritize fixes (TypeScript loading, summarization)
+3. Create issues for each problem found
+4. Re-run tests after fixes
+5. Add automated tests to prevent regressions
+
+---
+
+**Test Completed:** 2026-01-24
+**Tester:** Claude Sonnet 4.5
+**Status:** 10/12 tests passing, 2 critical issues found