memorylink 1.0.0

package/TREE_SITTER_EXPLANATION.md

@@ -0,0 +1,303 @@
+# Tree-sitter: What It Is and Why We Need It
+
+**Date**: December 15, 2025  
+**Status**: Explanation + Implementation Decision
+
+---
+
+## 🔍 WHAT IS TREE-SITTER?
+
+**Tree-sitter** is a parser generator tool that can parse source code in **multiple programming languages** into Abstract Syntax Trees (AST).
+
+### Key Features:
+1. **Multi-Language Support**: Python, Go, Rust, Ruby, Java, C++, PHP, and 40+ languages
+2. **Incremental Parsing**: Fast updates when code changes
+3. **Error Recovery**: Handles incomplete/broken code gracefully
+4. **Universal API**: Same interface for all languages
+
+---
+
+## 🎯 WHAT PURPOSE DOES IT SERVE?
+
+### Current MemoryLink Implementation:
+```typescript
+// Current: Only supports JavaScript/TypeScript
+if (filepath.endsWith('.ts') || filepath.endsWith('.tsx')) {
+  // Use TypeScript compiler API
+} else if (filepath.endsWith('.js') || filepath.endsWith('.jsx')) {
+  // Use regex parsing
+} else {
+  return null; // ❌ SKIPS Python, Go, Rust, etc.
+}
+```
+
+### With Tree-sitter:
+```typescript
+// Would support ALL languages
+if (filepath.endsWith('.py')) {
+  // Parse Python with tree-sitter-python
+} else if (filepath.endsWith('.go')) {
+  // Parse Go with tree-sitter-go
+} else if (filepath.endsWith('.rs')) {
+  // Parse Rust with tree-sitter-rust
+} else if (filepath.endsWith('.ts') || filepath.endsWith('.js')) {
+  // Parse TypeScript/JavaScript (can still use TS API or tree-sitter)
+}
+```
+
+---
+
+## ❓ WHY DO WE NEED IT?
+
+### Problem: Current Limitation
+
+**MemoryLink's code structure memory currently:**
+- ✅ Works for TypeScript/JavaScript projects
+- ❌ **SKIPS** Python files (`.py`)
+- ❌ **SKIPS** Go files (`.go`)
+- ❌ **SKIPS** Rust files (`.rs`)
+- ❌ **SKIPS** Ruby files (`.rb`)
+- ❌ **SKIPS** Java files (`.java`)
+- ❌ **SKIPS** C++ files (`.cpp`)
+
+### Real-World Impact:
+
+**Scenario 1: Python Project**
+```bash
+$ memorylink graph build
+📊 Building code dependency graph...
+   Files analyzed: 5        # ❌ Only found .ts files
+   Dependencies: 3
+   
+# But project has 50 Python files! ❌
+```
+
+**Scenario 2: Multi-Language Project**
+```
+project/
+├── backend/          # Python (Django)
+│   ├── models.py     # ❌ NOT ANALYZED
+│   └── views.py      # ❌ NOT ANALYZED
+├── frontend/         # TypeScript (React)
+│   ├── App.tsx       # ✅ ANALYZED
+│   └── utils.ts      # ✅ ANALYZED
+└── scripts/          # Go
+    └── build.go      # ❌ NOT ANALYZED
+```
+
+**Result**: Only 33% of codebase analyzed! ❌
+
+---
+
+## 🤔 WHY WASN'T IT DEVELOPED?
+
+### Reasons I Deferred It:
+
+1. **Current Implementation Covers Primary Use Case**
+   - Most MemoryLink users work with JavaScript/TypeScript
+   - Covers 90%+ of Node.js/React/Vue projects
+   - TypeScript compiler API is very accurate
+
+2. **Complexity vs. Value**
+   - Tree-sitter requires:
+     - Installing language-specific parsers (tree-sitter-python, tree-sitter-go, etc.)
+     - Handling different import/export syntax per language
+     - Testing across multiple languages
+   - Estimated effort: 1-2 weeks
+
+3. **Not Critical for v1.0 Launch**
+   - Core functionality works for JS/TS projects
+   - Can be added incrementally when needed
+   - Users can request it if they need it
+
+---
+
+## ✅ SHOULD WE IMPLEMENT IT?
+
+### Arguments FOR Implementation:
+
+1. **Future-Proofing**
+   - Many projects are multi-language
+   - Python is very popular (Django, Flask, ML projects)
+   - Go is growing (microservices, CLI tools)
+   - Rust is emerging (systems programming)
+
+2. **Competitive Advantage**
+   - Most tools only support one language
+   - Multi-language support = unique feature
+   - Appeals to larger, diverse codebases
+
+3. **User Requests**
+   - If users have Python/Go projects, they'll need it
+   - Better to have it now than add later
+
+### Arguments AGAINST (Current):
+
+1. **YAGNI Principle** (You Aren't Gonna Need It)
+   - Don't build features until needed
+   - Current users are JS/TS focused
+   - Can add when demand exists
+
+2. **Maintenance Burden**
+   - More languages = more edge cases
+   - More dependencies to maintain
+   - More testing required
+
+3. **Complexity**
+   - Different languages have different import systems
+   - Python: `import module` vs `from module import`
+   - Go: `import "package"` vs `import . "package"`
+   - Rust: `use crate::module` vs `use std::collections`
+
+---
+
+## 🎯 RECOMMENDATION
+
+### Option 1: Implement Now ✅ (Recommended if you have multi-language projects)
+
+**Pros:**
+- Complete feature set
+- Works for all codebases
+- Future-proof
+
+**Cons:**
+- 1-2 weeks development time
+- More dependencies
+- More complexity
+
+### Option 2: Keep Current (Recommended if JS/TS only)
+
+**Pros:**
+- Simple, focused
+- Fast and accurate for JS/TS
+- Less maintenance
+
+**Cons:**
+- Doesn't work for Python/Go/Rust projects
+- Users with multi-language projects can't use it
+
+---
+
+## 💻 IMPLEMENTATION PLAN (If We Do It)
+
+### Step 1: Install Tree-sitter
+```bash
+npm install tree-sitter tree-sitter-python tree-sitter-go tree-sitter-rust
+```
+
+### Step 2: Update parseFile()
+```typescript
+export function parseFile(filepath: string, rootDir: string): FileNode | null {
+  // TypeScript/JavaScript (current)
+  if (filepath.endsWith('.ts') || filepath.endsWith('.tsx')) {
+    return parseTypeScript(filepath, rootDir);
+  }
+  
+  // Python (NEW)
+  if (filepath.endsWith('.py')) {
+    return parsePython(filepath, rootDir);
+  }
+  
+  // Go (NEW)
+  if (filepath.endsWith('.go')) {
+    return parseGo(filepath, rootDir);
+  }
+  
+  // Rust (NEW)
+  if (filepath.endsWith('.rs')) {
+    return parseRust(filepath, rootDir);
+  }
+  
+  // ... more languages
+}
+```
+
+### Step 3: Language-Specific Parsers
+```typescript
+function parsePython(filepath: string, rootDir: string): FileNode | null {
+  const Parser = require('tree-sitter');
+  const Python = require('tree-sitter-python');
+  
+  const parser = new Parser();
+  parser.setLanguage(Python);
+  
+  const sourceCode = fs.readFileSync(filepath, 'utf-8');
+  const tree = parser.parse(sourceCode);
+  
+  // Extract imports: `import module`, `from module import`
+  // Extract functions: `def function_name`
+  // Extract classes: `class ClassName`
+  
+  return {
+    path: normalizePathForGraph(filepath, rootDir),
+    imports: extractPythonImports(tree),
+    exports: [], // Python doesn't have explicit exports
+    functions: extractPythonFunctions(tree),
+    classes: extractPythonClasses(tree),
+    // ...
+  };
+}
+```
+
+### Step 4: Update findSourceFiles()
+```typescript
+const sourceFiles = await glob([
+  '**/*.ts',
+  '**/*.tsx',
+  '**/*.js',
+  '**/*.jsx',
+  '**/*.py',    // NEW
+  '**/*.go',    // NEW
+  '**/*.rs',    // NEW
+  // ... more patterns
+], { cwd: rootDir, ignore: ['node_modules/**', '.git/**'] });
+```
+
+---
+
+## 📊 DECISION MATRIX
+
+| Factor | Implement Now | Defer |
+|--------|---------------|-------|
+| **Current Users** | JS/TS only | JS/TS only |
+| **Future Users** | All languages | JS/TS only |
+| **Development Time** | 1-2 weeks | 0 weeks |
+| **Maintenance** | Higher | Lower |
+| **Competitive Edge** | High | Medium |
+| **User Requests** | None yet | Can add when requested |
+
+---
+
+## 🎯 MY RECOMMENDATION
+
+**If you have Python/Go/Rust projects**: ✅ **Implement it now**
+- You'll need it eventually
+- Better to have complete feature
+- 1-2 weeks is reasonable
+
+**If you only have JS/TS projects**: ⚠️ **Defer it**
+- Current implementation is sufficient
+- Can add when needed
+- Focus on other features first
+
+---
+
+## ❓ YOUR DECISION
+
+**Do you want me to implement Tree-sitter support now?**
+
+If yes, I'll:
+1. Add tree-sitter dependencies
+2. Implement Python parser
+3. Implement Go parser
+4. Implement Rust parser
+5. Update graph building to include all languages
+6. Test with multi-language projects
+
+**Estimated time**: 1-2 weeks  
+**Impact**: High (enables multi-language projects)
+
+---
+
+**What do you think? Should we implement it now or defer it?**
+

package/TROUBLESHOOTING.md

@@ -0,0 +1,62 @@
+# MemoryLink Troubleshooting Guide
+
+This document helps you debug common MemoryLink issues.
+
+## 1. Installation Issues
+- **Symptom**: `memorylink` command not found.
+- **Fix**:
+  - Run: `npm install -g memorylink` OR use `npx memorylink`.
+  - Confirm: `memorylink --version`.
+
+## 2. Project Not Initialized
+- **Symptom**: Errors like `MemoryLink is not initialized. Run: memorylink init`.
+- **Fix**:
+  - From your project root, run: `memorylink init`.
+  - This creates the `.memorylink` folder and default config.
+
+## 3. Git Hooks / Auto-Log Not Working
+- **Symptom**: Commits are not logged to `changes.log`.
+- **Fix**:
+  - Run: `memorylink hooks install`.
+  - Verify: `memorylink hooks status`.
+  - Ensure Git is installed and the project is a Git repo.
+
+## 4. get-context Slow or Too Large
+- **Symptom**: `get-context` feels slow or returns huge output.
+- **Fix**:
+  - Use `--limit` and `--tag` flags to narrow results.
+  - Example: `memorylink get-context --limit 20 --tag auth`.
+  - Consider archiving old entries if logs are extremely large.
+
+## 5. Vector Search Not Working
+- **Symptom**: Errors about `OPENAI_API_KEY` or empty vector results.
+- **Fix**:
+  - For OpenAI-based search:
+    - Set `OPENAI_API_KEY` in your environment.
+    - Run: `memorylink index --vector`.
+  - For local embeddings:
+    - Install: `npm install @xenova/transformers`.
+    - Run: `memorylink index --vector --local`.
+
+## 6. Code Graph / Impact Issues
+- **Symptom**: `graph build` fails or impact results look wrong.
+- **Fix**:
+  - Run: `memorylink graph build --rebuild`.
+  - Ensure your project compiles and there are no syntax errors.
+
+## 7. Using the doctor Command
+- **Symptom**: Something feels off and you’re not sure why.
+- **Fix**:
+  - Run: `memorylink doctor`.
+  - This will check:
+    - Project initialization
+    - Log file health
+    - Git integration
+    - Code graph freshness
+  - Follow the suggestions printed by `doctor`.
+
+If issues persist, open a GitHub issue with:
+- Your Node.js version
+- MemoryLink version
+- OS
+- The exact command and output