encoding-aware-fs 0.1.3 → 0.1.5

package/README.md

@@ -0,0 +1,54 @@
+# encoding-aware-fs
+
+面向 AI 编码工具（Claude Code、OpenCode 等）的编码感知文件操作 MCP 服务器，专为处理 GB18030/GBK/GB2312 编码项目而设计。
+
+## 问题背景
+
+AI 编码助手默认所有文件都是 UTF-8 编码，在读写 GB18030/GBK/GB2312 编码的文件时会导致静默乱码——这在中文遗留项目中非常常见。本 MCP 服务器透明地处理编码检测与转换，让 AI 工具能正确操作非 UTF-8 文件。
+
+## 功能特性
+
+- **自动编码检测** — 读取文件时自动识别原始编码（GB18030/GBK/GB2312），返回 UTF-8 给 AI 工具
+- **透明回写** — 写入时自动将 UTF-8 内容转换回文件的原始编码
+- **行尾保持** — 检测并保持文件原有的 CRLF/LF 行尾格式，编辑后不会改变
+- **三个 MCP 工具**：`read_file`、`write_file`、`edit_file` — 替代内置文件操作工具
+- **一键安装/卸载** — 支持 Claude Code 和 OpenCode
+
+## 安装
+
+```bash
+npx encoding-aware-fs install
+```
+
+该命令会在项目的 `.mcp.json`（Claude Code）或 `.opencode.json`（OpenCode）中注册 MCP 服务器，并添加规则文件指示 AI 使用 MCP 工具进行所有文件操作。
+
+## 卸载
+
+```bash
+npx encoding-aware-fs uninstall
+```
+
+## MCP 工具
+
+| 工具 | 说明 |
+|------|------|
+| `read_file` | 读取文件，自动检测编码并返回 UTF-8 内容 |
+| `write_file` | 写入 UTF-8 内容，自动转换为文件原始编码 |
+| `edit_file` | 查找替换编辑，保持编码和行尾格式 |
+
+## 工作原理
+
+1. **读取时**：通过 BOM 和启发式算法检测文件编码，解码为 UTF-8 返回
+2. **写入/编辑时**：先读取原文件检测编码和行尾风格，在 UTF-8 下完成操作，写入前转回原编码并还原行尾格式
+
+## 开发
+
+```bash
+npm install
+npm run build
+npm test
+```
+
+## 许可证
+
+MIT

package/RESEARCH.md

@@ -0,0 +1,502 @@
+# Encoding-Aware-FS MCP Project Research
+
+**Date**: 2026-03-26
+**Project Location**: F:\WorkSpace\projects\filesystemex
+**Version**: 0.1.4
+
+## Executive Summary
+
+The **encoding-aware-fs** is a sophisticated MCP (Model Context Protocol) server designed to handle file operations with transparent encoding detection and conversion. It specializes in working with GB18030/GBK/GB2312 encoded files while maintaining UTF-8 compatibility for AI tools.
+
+---
+
+## 1. Overall Project Structure
+
+### Directory Layout
+```
+filesystemex/
+├── .git/                          # Git repository
+├── .claude/                       # Claude-specific config
+├── .agents/                       # Agent definitions
+├── src/                           # TypeScript source code
+│   ├── server.ts                  # Main MCP server entry
+│   ├── index.ts                   # CLI router (install/uninstall/serve)
+│   ├── tools/                     # MCP Tool implementations
+│   │   ├── read-file.ts          # Read file tool with encoding detection
+│   │   ├── write-file.ts         # Write file tool with encoding preservation
+│   │   └── edit-file.ts          # Edit file tool (THE KEY TOOL)
+│   ├── encoding/                  # Encoding utilities
+│   │   ├── detector.ts           # Python chardet integration
+│   │   ├── converter.ts          # iconv-lite encoding/decoding
+│   │   ├── line-endings.ts       # CRLF/LF detection and restoration
+│   │   └── binary.ts             # Binary file detection
+│   ├── config.ts                  # Project configuration loader
+│   ├── path-validation.ts         # Security path validation with symlink checks
+│   ├── path-utils.ts             # Path utilities (normalization, home expansion)
+│   ├── config-io.ts              # Config file I/O
+│   ├── logger.ts                 # Structured logging
+│   ├── installer.ts              # One-click installation
+│   └── uninstaller.ts            # One-click uninstallation
+├── dist/                          # Compiled JavaScript
+│   ├── server.js                 # Bundled MCP server
+│   └── index.js                  # Bundled CLI entry point
+├── tests/                         # Vitest test suite
+│   ├── fixtures/                 # Test files (GB18030, UTF-8, binary)
+│   └── encoding/                 # Encoding tests
+├── docs/                          # Documentation
+├── skills/                        # Claude Code skills
+├── package.json                   # NPM package metadata
+├── tsconfig.json                  # TypeScript configuration
+├── build.js                       # esbuild build script
+├── .mcp.json                      # MCP server configuration
+├── .encoding-converter.json       # Project encoding config
+└── README.md                      # Project documentation
+```
+
+### Key Dependencies
+- `@modelcontextprotocol/sdk` - MCP protocol implementation
+- `zod` - Schema validation
+- `iconv-lite` - Encoding conversion
+- `diff` - Unified diff generation
+- `@inquirer/prompts` - Interactive CLI
+
+---
+
+## 2. The Edit_File Tool - Complete Implementation
+
+### 2.1 What edit_file Returns
+
+The `edit_file` tool returns a **unified diff** showing all changes:
+
+```typescript
+return {
+  content: [{ type: "text" as const, text: diff }]
+}
+```
+
+Example return format:
+```
+--- /path/to/file.txt
++++ /path/to/file.txt
+@@ -10,3 +10,3 @@
+ context line
+-removed text
++new text
+ context line
+```
+
+### 2.2 Tool Input Schema
+
+```typescript
+{
+  path: z.string().describe("Absolute file path"),
+  edits: z.array(z.object({
+    oldText: z.string().describe("Text to search for"),
+    newText: z.string().describe("Text to replace with"),
+  })),
+  dryRun: z.boolean().default(false).describe("Preview changes without writing"),
+}
+```
+
+### 2.3 Handler Execution Flow (9 Steps)
+
+**Step 1: Path Validation**
+- Validates path is within allowed directories
+- Resolves symlinks to prevent escape attacks
+- For new files, validates parent directory exists
+
+**Step 2: Read Raw Bytes**
+- Reads file as binary buffer
+- Preserves original line endings
+
+**Step 3: Detect Line Ending Style**
+- Examines raw bytes for CRLF vs LF
+- Returns 'CRLF' or 'LF'
+- Ties favor CRLF
+
+**Step 4: Detect Encoding**
+- Spawns Python process with chardet library
+- Reads first 4KB of file
+- Returns encoding and confidence (0-1)
+- Uses confidence threshold (default 0.8)
+- Falls back to project config sourceEncoding
+
+**Step 5: Decode to UTF-8**
+- Uses iconv-lite to convert detected encoding to UTF-8
+- All further processing uses UTF-8
+
+**Step 6: Apply Edits**
+- Normalizes line endings to LF for comparison
+- Tries exact string match first
+- Falls back to fuzzy line-by-line matching
+- Fuzzy matching compares trimmed lines, preserves indentation
+- Throws error if no match found
+
+**Step 7: Generate Diff**
+- Uses `diff` package's `createTwoFilesPatch()` function
+- Generates unified diff format
+- Normalizes line endings to LF for comparison
+
+**Step 8: Conditional Write**
+- If dryRun=true: returns diff only
+- If dryRun=false:
+  - Restores original line ending style
+  - Encodes back to original detected encoding
+  - Performs atomic write (temp file → rename)
+
+**Step 9: Return Result**
+- Returns standardized MCP response with diff text
+
+---
+
+## 3. Fuzzy Edit Matching Algorithm
+
+Location: `src/tools/edit-file.ts`, lines 25-74
+
+Two-phase matching strategy:
+
+**Phase 1: Exact Match**
+```
+Search for oldText substring in content
+If found, replace with newText
+```
+
+**Phase 2: Fuzzy Line-by-Line Match**
+```
+For each position in content:
+  Compare trimmed lines with oldText lines
+  If all lines match (trimmed):
+    - Get original indentation from first line
+    - Apply newText preserving indentation
+    - Handle relative indentation for multi-line replacements
+    - Continue to next edit
+  If no match found:
+    Throw "Could not find match for edit" error
+```
+
+Key features:
+- Whitespace-tolerant
+- Indentation-aware
+- Relative indentation handling
+- Deterministic (uses first match)
+
+---
+
+## 4. MCP Server Communication
+
+### 4.1 Server Startup
+
+```typescript
+// 1. Create McpServer instance
+const server = new McpServer({
+  name: "encoding-aware-fs",
+  version: "0.1.0",
+})
+
+// 2. Load configuration from .encoding-converter.json
+config = await loadProjectConfig(process.cwd())
+
+// 3. Define allowed directories
+const allowedDirectories = [process.cwd()]
+
+// 4. Register tools
+registerReadFile(server, config, allowedDirectories)
+registerWriteFile(server, config, allowedDirectories)
+registerEditFile(server, config, allowedDirectories)
+
+// 5. Connect stdio transport
+const transport = new StdioServerTransport()
+await server.connect(transport)
+```
+
+### 4.2 Communication Protocol
+
+**Transport**: stdio (stdin/stdout)
+
+**Message Flow**:
+1. Client sends JSON-RPC request via stdin
+2. Server parses and validates with Zod schema
+3. Server executes async handler
+4. Handler returns result object
+5. Server wraps in JSON-RPC response
+6. Server writes to stdout
+7. Client parses and displays result
+
+### 4.3 Response Format
+
+Standard MCP response:
+
+```typescript
+{
+  content: [
+    {
+      type: "text",     // Can also be "image" or "resource"
+      text: string      // The actual content
+    }
+  ]
+}
+```
+
+---
+
+## 5. Encoding Handling
+
+### 5.1 Encoding Detection
+
+Uses Python's `chardet` library:
+
+```typescript
+async function detectEncoding(filePath: string): Promise<DetectionResult> {
+  // Spawns Python process running chardet.detect()
+  // Reads first 4KB of file
+  // Returns { encoding: string | null, confidence: number }
+}
+```
+
+Supported encodings:
+- GB18030 (default for legacy projects)
+- GBK
+- GB2312
+- UTF-8
+- Any encoding supported by iconv-lite
+
+### 5.2 Encoding Conversion
+
+Uses iconv-lite:
+
+```typescript
+// Decoding: Any encoding → UTF-8
+function decodeToUtf8(buffer: Buffer, encoding: string): string {
+  return iconv.decode(buffer, encoding)
+}
+
+// Encoding: UTF-8 → Target encoding
+function encodeFromUtf8(text: string, encoding: string): Buffer {
+  return iconv.encode(text, encoding)
+}
+```
+
+### 5.3 Line Ending Preservation
+
+Detects and preserves CRLF vs LF:
+
+```typescript
+function detectLineEnding(buffer: Buffer): 'CRLF' | 'LF' {
+  // Count 0x0D 0x0A (CRLF) vs standalone 0x0A (LF)
+  // Returns whichever is dominant
+}
+
+function restoreLineEndings(text: string, style: 'CRLF' | 'LF'): string {
+  // Normalize to LF first, then convert to target style
+}
+```
+
+---
+
+## 6. Security: Path Validation
+
+Location: `src/path-validation.ts`
+
+Multi-layer security checks:
+
+1. Type validation
+2. Null byte prevention (path injection)
+3. Absolute path enforcement
+4. Allowed directory boundary check
+5. Symlink resolution & re-validation
+6. Parent directory validation (for new files)
+
+Prevents:
+- Directory traversal attacks
+- Symlink escape attacks
+- Writing outside project boundaries
+
+---
+
+## 7. Atomic File Write Pattern
+
+Used by both write_file and edit_file:
+
+```typescript
+const tempPath = `${validPath}.${randomBytes(16).toString('hex')}.tmp`
+try {
+  await fs.writeFile(tempPath, encoded)
+  await fs.rename(tempPath, validPath)  // Atomic operation
+} catch (error) {
+  try { await fs.unlink(tempPath) } catch {}  // Cleanup
+  throw error
+}
+```
+
+Benefits:
+- Prevents partial writes on crash
+- Atomic rename prevents corruption
+- Automatic cleanup on error
+
+---
+
+## 8. Diff Generation
+
+Location: `src/tools/edit-file.ts`, lines 16-18
+
+```typescript
+function createUnifiedDiff(
+  original: string,
+  modified: string,
+  filepath: string
+): string {
+  return createTwoFilesPatch(
+    filepath,
+    filepath,
+    normalizeLineEndings(original),
+    normalizeLineEndings(modified),
+    'original',
+    'modified'
+  )
+}
+```
+
+Uses `diff` package to generate standard unified diff format.
+
+---
+
+## 9. Logging
+
+Location: `src/logger.ts`
+
+Structured logging with configurable levels:
+
+```typescript
+function log(
+  level: 'debug' | 'info' | 'error',
+  message: string,
+  data?: object
+): void
+```
+
+- Logs to: `~/.encoding-aware-fs/logs/server.log`
+- Level controlled by: `ENCODING_CONVERTER_LOG` env variable
+- Default level: 'error'
+- Includes timestamps and JSON data
+
+Used to track encoding detection and file operations.
+
+---
+
+## 10. Configuration
+
+### .encoding-converter.json
+
+```json
+{
+  "sourceEncoding": "GB18030",
+  "targetEncoding": "UTF-8",
+  "confidenceThreshold": 0.8
+}
+```
+
+Controls:
+- Default encoding for undetected files
+- Encoding detection confidence threshold
+- Target output encoding
+
+### .mcp.json
+
+```json
+{
+  "mcpServers": {
+    "encoding-aware-fs": {
+      "command": "cmd",
+      "args": ["/c", "npx", "-y", "encoding-aware-fs", "serve"],
+      "env": {}
+    }
+  }
+}
+```
+
+Registers the MCP server with Claude Code.
+
+---
+
+## 11. Data Processing Flow
+
+```
+User Input (AI Tool)
+    ↓
+[stdin] JSON-RPC Request
+    ↓
+Path Validation (security checks)
+    ↓
+Read raw bytes
+    ↓
+Detect line ending (CRLF vs LF)
+    ↓
+Detect encoding (Python chardet)
+    ↓
+Decode to UTF-8 (iconv-lite)
+    ↓
+Apply edits (exact or fuzzy matching)
+    ↓
+Generate unified diff
+    ↓
+DryRun?
+  YES → Return diff only
+  NO  → Restore line endings → Encode to original → Atomic write
+    ↓
+[stdout] JSON-RPC Response with diff
+    ↓
+Client displays diff
+```
+
+---
+
+## 12. Testing
+
+Framework: Vitest
+
+Test files:
+- `tests/encoding/line-endings.test.ts`
+- `tests/encoding/detector.test.ts`
+- `tests/encoding/converter.test.ts`
+- `tests/encoding/binary.test.ts`
+- `tests/config.test.ts`
+
+Run tests:
+```bash
+npm test          # Run once
+npm run test:watch # Watch mode
+npm run typecheck  # TypeScript validation
+```
+
+---
+
+## 13. Summary
+
+### What edit_file Returns
+
+A **unified diff** in standard format, showing:
+- File paths and timestamps
+- Line numbers and change counts
+- Context lines (unchanged)
+- Removed lines (prefixed with `-`)
+- Added lines (prefixed with `+`)
+
+### Communication Model
+
+1. **Input**: JSON-RPC call with path and edits array
+2. **Processing**: Detect encoding → decode → apply edits → generate diff → optionally write
+3. **Output**: MCP response with diff text
+4. **Optional**: Write changes to disk (atomic, preserves encoding/line-endings)
+
+### Unique Features
+
+1. **Transparent Encoding Handling**: Works with legacy GB18030 projects
+2. **Line Ending Preservation**: Maintains original CRLF vs LF
+3. **Smart Matching**: Fuzzy matching handles indentation variations
+4. **Safe Operations**: Atomic writes prevent corruption
+5. **DryRun Support**: Preview before committing
+6. **Security**: Comprehensive path validation and symlink checks
+
+---
+
+**End of Research Document**