pi-charset 0.0.0-stage → 0.1.2

package/CHANGELOG.md

@@ -0,0 +1,70 @@
+# Changelog
+
+English | [简体中文](./CHANGELOG.zh-CN.md)
+
+All notable changes to this project will be documented in this file.
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+---
+
+## [0.1.2] - 2026-10-28
+
+### Added
+
+- Add in-session `/charset-init` and `/charset-scan` scaffolding commands (`src/scaffold.ts`, `src/index.ts`) supporting scan-based `.pi-charset.json` generation, preset templates (`embedded-c`, `legacy-java-web`, `mixed-repo`, `all-utf8`), `--dry-run` preview, `--force` overwrite control, and immediate session cache refresh.
+- Bundle the `charset-rules` Agent Skill (`skills/charset-rules/SKILL.md`) and `/charset-setup` prompt template (`prompts/charset-setup.md`) in the Pi package manifest (`package.json` `pi.skills` and `pi.prompts`).
+- Add automated release verification (`tools/check-release.mjs`), `npm version` synchronization (`tools/sync-version.mjs`), changelog section extraction (`tools/extract-changelog.mjs`), tag-triggered GitHub Release & optional npm provenance workflow (`.github/workflows/release.yml`), and repository engineering rules (`AGENTS.md`, `.editorconfig`, `CONTRIBUTING.md`, `docs/RELEASING.md`).
+
+### Changed
+
+- Split all English and Simplified Chinese documentation into standalone, cross-linked files (`README.md` / `README.zh-CN.md`, `CHANGELOG.md` / `CHANGELOG.zh-CN.md`, `CONTRIBUTING.md` / `CONTRIBUTING.zh-CN.md`, `docs/*.md` / `docs/zh-CN/*.md`).
+
+## [0.1.1] - 2026-10-27
+
+### Fixed
+
+- Preserve literal question marks when escaping characters that the target encoding cannot represent; use encoding round trips rather than replacement-byte heuristics.
+- Detect UTF-16 line endings at the character level, including characters whose bytes resemble CR or LF.
+- Check destination write access before atomic replacement so read-only files remain protected on POSIX systems.
+- Keep path keys case-sensitive outside Windows and reset classification caches when the extension reloads.
+- Load the optional ripgrep package through an ESM-compatible module loader; deduplicate search results only after validation.
+- Apply line-ending guards to UTF-8 writes as well as legacy encodings; detect deletion after a read and recheck destination content before each rename attempt.
+- Leave regex matching to ripgrep, including ASCII patterns such as `\p{Han}` that can match non-ASCII text.
+- Validate shell fallback encodings before constructing a streaming decoder.
+
+### Changed
+
+- Reuse per-character encoding checks within each conversion, and reuse the prepared temporary file during Windows rename retries instead of rewriting it.
+- Preserve existing file modes during atomic replacement and reject writes to symbolic links instead of replacing the link itself.
+- Add GBK setup guidance and document the permanent content changes caused by `escape`, as well as the I/O cost of write verification.
+- Add Node 24 to the CI matrix and make strict I/O timing checks opt-in with `STRICT_PERF=1`.
+- Update Vitest and the dependency lockfile; pin the checkout's transitive `undici` to a patched range. Overrides do not update a separately installed Pi host.
+
+## [0.1.0] - 2026-10-27
+
+### Added
+
+- **Deterministic Byte Classifier** (`src/encoding/classify.ts`)
+  - In-memory classification pipeline powered by Node native C++ SIMD (`node:buffer` `isAscii` / `isUtf8`): `BOM -> NUL binary check -> force rule -> pure ASCII -> strict UTF-8 -> CJK lossless round-trip -> config fallback`.
+  - Zero-config self-adaptive mode with same-directory sibling inference for newly created and pure-ASCII files.
+  - File-level classification cache keyed by `(path, mtimeMs, size, configKey)`.
+
+- **Write Safety Gates & Conflict Detection** (`src/operations.ts`, `src/resolve.ts`)
+  - Gate 1: Unmappable code point detection supporting `error`, `escape` (`\uXXXX`), and `drop-to-gb18030` strategies.
+  - Gate 2: Pre-write in-memory round-trip verification and post-write disk read-back verification, preserving on-disk state on conflict without rolling back over external changes.
+  - Gate 3: Rejects writes to `BINARY` and `UNKNOWN` files, blocks writing `U+FEFF` BOM into GB-family encodings, and blocks lossy re-encoding of existing files.
+  - SHA-1 read baseline tracking to detect external file modifications between `read` and `write`.
+  - Per-file serialized write locks and atomic temp-file `fsync + rename` replacement with Windows lock retries.
+
+- **Line-Ending Handling** (`src/encoding/line-endings.ts`)
+  - Preserves existing `CRLF` or `LF` line endings; allows reading `MIXED` (`CRLF` + `LF`) files while rejecting `edit` and `write`.
+  - Supports `newFileLineEnding: "CRLF" | "LF"` for newly created files.
+
+- **Multi-Encoding Search & Edit Preview** (`src/grep.ts`, `src/edit-preview.ts`)
+  - Multi-pass `ripgrep` execution across configured encodings with per-file classification verification and `file:line` deduplication.
+  - Encoding-aware `edit` diff preview for non-UTF-8 managed files, delegating UTF-8 files to Pi's built-in renderer.
+
+- **Configuration, Scanner CLI & Diagnostics** (`src/config.ts`, `tools/scan-encoding.mjs`, `src/index.ts`)
+  - Configuration file `.pi-charset.json` with JSONC comment support.
+  - Standalone scanner `tools/scan-encoding.mjs` supporting directory profiling, `--damaged` inspection, and `--init` config generation.
+  - Diagnostic command `/charset-status`.

package/CHANGELOG.zh-CN.md

@@ -0,0 +1,69 @@
+# 版本变更日志
+
+[English](./CHANGELOG.md) | 简体中文
+
+本文件记录项目的重要版本变更，格式遵循 [Keep a Changelog](https://keepachangelog.com/zh-CN/1.1.0/) 与[语义化版本](https://semver.org/lang/zh-CN/)。
+
+---
+
+## [0.1.2] - 2026-10-28
+
+### 新增
+
+- 新增会话内 `/charset-init` 与 `/charset-scan` 脚手架命令（`src/scaffold.ts`、`src/index.ts`），支持工程自动扫描生成 `.pi-charset.json`、预设模板套用（`embedded-c`、`legacy-java-web`、`mixed-repo`、`all-utf8`）、`--dry-run` 预览、`--force` 覆盖控制与当前会话缓存即时刷新。
+- 在 `package.json` 的 `pi` 清单中随包分发 `charset-rules` 编码安全规则 Skill（`skills/charset-rules/SKILL.md`）与 `/charset-setup` 引导式提示模板（`prompts/charset-setup.md`）。
+- 新增发布门禁脚本（`tools/check-release.mjs`）、`npm version` 版本同步钩子（`tools/sync-version.mjs`）、发布说明提取脚本（`tools/extract-changelog.mjs`）、标签触发的 GitHub Release 与可选 npm Provenance 自动发布工作流（`.github/workflows/release.yml`），以及仓内工程规范（`AGENTS.md`、`.editorconfig`、`CONTRIBUTING.zh-CN.md`、`docs/zh-CN/RELEASING.md`）。
+
+### 调整
+
+- 将全部中英文文档拆分为独立互链文件（`README.md` / `README.zh-CN.md`、`CHANGELOG.md` / `CHANGELOG.zh-CN.md`、`CONTRIBUTING.md` / `CONTRIBUTING.zh-CN.md`、`docs/*.md` / `docs/zh-CN/*.md`）。
+
+## [0.1.1] - 2026-10-27
+
+### 修复
+
+- 修复不可映射字符转义时误改合法问号的问题，改用编解码回环判断字符是否可表示。
+- 按字符识别 UTF-16 换行，避免将普通字符中的字节误认成行尾。
+- 原子替换前检查目标文件写权限，避免在 POSIX 系统上绕过只读保护。
+- 非 Windows 平台保留路径大小写，扩展重载时一并清理编码判定缓存。
+- 修复 ESM 环境下的可选 ripgrep 加载，在验证命中可信后再去重。
+- UTF-8 写入同样执行行尾保护；检测读取后的外部删除，并在每次重命名前复核目标内容。
+- 正则匹配交由 ripgrep 处理，修复 `\p{Han}` 等纯 ASCII 模式漏搜传统编码文本的问题。
+- 校验 Shell 回退编码，避免不支持的编码名导致流式解码异常。
+
+### 调整
+
+- 单次转换内复用重复字符的编码判定；Windows 重命名重试复用已写好的临时文件，避免重复写盘。
+- 原子替换时保留已有文件权限；符号链接拒绝写入，不再直接替换链接本身。
+- 补充 GBK 工程使用步骤，说明 `escape` 的永久内容改写及写后校验的 I/O 成本。
+- CI 增加 Node 24；严格 I/O 耗时断言改为通过 `STRICT_PERF=1` 启用。
+- 更新 Vitest 和依赖锁文件，将本仓库的间接依赖 `undici` 限定到已修复范围；此覆盖规则不会升级另行安装的 Pi 宿主。
+
+## [0.1.0] - 2026-10-27
+
+### 新增
+
+- **确定性字节编码判定**（`src/encoding/classify.ts`）
+  - 基于 Node 原生 C++ SIMD（`node:buffer` `isAscii` / `isUtf8`）的纯内存编码判定链：`BOM -> NUL 二进制检查 -> force 规则 -> 纯 ASCII -> 严格 UTF-8 -> CJK 多字节无损回环 -> 配置兜底`。
+  - 零配置自适应模式：已有文件自动保持原编码与行尾，新建文件与纯 ASCII 文件自动根据同目录邻居文件推断编码与行尾。
+  - 基于 `(path, mtimeMs, size, configKey)` 的文件级判定缓存。
+
+- **写入门禁与冲突检测**（`src/operations.ts`、`src/resolve.ts`）
+  - 闸门 1：目标编码不可映射字符检查，支持 `error`、`escape`（`\uXXXX`）与 `drop-to-gb18030` 三种策略。
+  - 闸门 2：写前内存编解码往返校验与写后磁盘回读核对；校验不一致时报错并保留磁盘现状，不执行旧备份回滚。
+  - 闸门 3：拦截 `BINARY`、`UNKNOWN`、BOM 写入 GB 族编码以及原文件字节无法无损回环的写入。
+  - 记录受管文件读取时的 SHA-1 基线指纹，在写入前检测外部进程修改冲突。
+  - 路径级串行写锁，配合同目录临时文件 `fsync + rename` 原子替换与 Windows 瞬时文件锁退避重试。
+
+- **行尾处理**（`src/encoding/line-endings.ts`）
+  - 保留已存在文件的 `CRLF` 或 `LF` 行尾；混合换行（`CRLF` 与 `LF` 并存）文件允许读取、拒绝写入。
+  - 支持通过 `newFileLineEnding: "CRLF" | "LF"` 配置新建文件的行尾格式。
+
+- **多编码搜索与编辑预览**（`src/grep.ts`、`src/edit-preview.ts`）
+  - `grep` 按配置编码集合分趟调用 `ripgrep`，经逐文件字节判定过滤乱码伪命中并按 `file:line` 去重。
+  - 针对非 UTF-8 受管文件提供编码感知的 `edit` diff 预览，UTF-8 文件直接委托内置渲染器。
+
+- **配置、扫描工具与诊断命令**（`src/config.ts`、`tools/scan-encoding.mjs`、`src/index.ts`）
+  - 标准配置文件 `.pi-charset.json`（支持 JSONC 注释）。
+  - 独立扫描脚本 `tools/scan-encoding.mjs`，支持目录编码画像、`--damaged` 受损文件检查与 `--init` 配置生成。
+  - 注册 `/charset-status` 诊断命令。

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 ethan-blue
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,3 +1,163 @@
-# Temporary Holding Version
+# pi-charset
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+[![CI](https://github.com/ethan-blue/pi-charset/actions/workflows/ci.yml/badge.svg)](https://github.com/ethan-blue/pi-charset/actions/workflows/ci.yml)
+[![Version](https://img.shields.io/badge/version-0.1.2-blue.svg)](https://github.com/ethan-blue/pi-charset/releases)
+[![Node](https://img.shields.io/badge/node-%3E%3D22.19.0-brightgreen.svg)](./package.json)
+[![License: MIT](https://img.shields.io/badge/license-MIT-yellow.svg)](./LICENSE)
+
+Encoding-aware `read`, `write`, `edit`, and `grep` extension for [Pi](https://github.com/earendil-works) (`@earendil-works/pi-coding-agent`).
+
+English | [简体中文](./README.zh-CN.md)
+
+---
+
+## Overview
+
+`pi-charset` intercepts file I/O and search operations in Pi to handle repositories that mix `GBK`, `GB18030`, `Big5`, `Shift_JIS`, `EUC-KR`, `ISO-8859-1`, `UTF-16`, and `UTF-8` files within the same workspace. All byte classification and transcoding run in-process without external Python or `chardet` subprocesses.
+
+## Working with a GBK Project
+
+For an existing GBK project, run `/charset-init --dry-run` (or `/charset-init embedded-c --dry-run`) inside Pi to review the suggested rules before generating `.pi-charset.json`. Keep `unmappable: "error"` and `verifyWrite: true`: characters that GBK cannot represent should stop a write rather than turn into `?`.
+
+Use Pi's `read`, `edit`, and `write` tools for source changes. Shell redirection and external formatters bypass this extension. After installation, run `/charset-status`, edit a small tracked file, and inspect the diff in your usual editor before wider use.
+
+Byte detection is not proof of an encoding: GBK and GB18030 overlap, and some bytes also decode under other encodings. For a project that requires GBK, declare GBK explicitly instead of relying on inference, especially for new or ASCII-only files. This extension prevents new encoding damage; it cannot reconstruct text that was already saved as question marks or mojibake.
+
+## Architecture
+
+### 1. Deterministic Byte Classification
+
+Files are classified in memory in a fixed order:
+1. **BOM Detection**: `UTF-8 BOM`, `UTF-16LE`, `UTF-16BE` (`UTF-32 BOM` is classified as `BINARY`).
+2. **Binary Guard**: Any `0x00` byte within the first 8192 bytes classifies the file as `BINARY`.
+3. **Explicit `force`**: Rules with `force: true` apply the configured encoding directly.
+4. **Pure ASCII**: Validated via Node's native `node:buffer` `isAscii` (`<= 0x7F`).
+5. **Strict UTF-8**: Full-buffer validation via Node's native `node:buffer` `isUtf8` without heap string allocation.
+6. **Multi-Byte CJK Round-Trip**: Candidates in `autoCandidates` are tested for `encode(decode(bytes)) == bytes` with zero `U+FFFD` replacement characters; `sourceEncoding` takes priority when multiple candidates pass.
+7. **Fallback**: Unclassified byte sequences are marked `UNKNOWN`.
+
+### 2. Write Safety Gates
+
+- **Gate 1 (Unmappable Code Points)**: Checks every code point against the target encoding before writing. Default policy `error` aborts the write and lists unmappable characters with `U+XXXX` code points. Optional policies: `escape` (`\uXXXX`) and `drop-to-gb18030` (promotes GB-family writes to `GB18030`).
+- **Gate 2 (Pre-Write Round-Trip & Post-Write Read-Back)**: Verifies `decode(encode(text)) === text` in memory before touching disk, then reads back the written file to verify byte and text equality. If post-write verification fails, `pi-charset` reports the mismatch and preserves the on-disk file without rolling back to a stale backup.
+- **Gate 3 (Unknown & Lossy Conversion Guard)**: Rejects writes to `BINARY` or `UNKNOWN` files, preserves existing `UTF-8` files by default (`protectUtf8: true`), and rejects writes when existing on-disk bytes cannot losslessly round-trip in the target encoding.
+
+### 3. Line Endings & Baseline Conflict Detection
+
+- **Line-Ending Preservation**: Existing files keep their on-disk line endings (`CRLF` or `LF`, detected at the character level for UTF-16); files without newlines do not get trailing newlines appended.
+- **Mixed Line-Ending Guard**: Files containing both `CRLF` and `LF` can be read via `read`, but `edit` and `write` are rejected.
+- **New-File Line Endings**: Configurable via `newFileLineEnding: "CRLF" | "LF"` for newly created files.
+- **Read Baseline Verification**: Records a SHA-1 fingerprint when a managed file is read; rejects writes if the on-disk file was modified or deleted externally after the read.
+
+### 4. Zero-Config Self-Adaptive Mode & Sibling Inference
+
+`pi-charset` works out-of-the-box without requiring a `.pi-charset.json` file:
+- **Existing Files**: Automatically read in their byte-classified encoding (`GBK`, `GB18030`, `Big5`, `UTF-16`, `UTF-8`) and written back preserving their original encoding, line endings (`CRLF` or `LF`), and BOM.
+- **New & Pure-ASCII Files**: When no `.pi-charset.json` is present, `pi-charset` inspects sibling files in the same directory (matching extension group first, e.g. `.c`/`.h`) and inherits the dominant encoding and line-ending style (defaulting to `UTF-8` when siblings are UTF-8 or ASCII).
+- **Multi-Encoding Search (`grep`)**: Regex patterns and non-ASCII literals run `UTF-8 + GB18030` `ripgrep` passes in zero-config mode, or the configured candidate encodings when `.pi-charset.json` is present. Matches are verified against each file's classified encoding and deduplicated by `file:line`.
+
+## Quick Start
+
+### 1. Installation
+
+```bash
+# Install from Git (-l writes to project .pi/settings.json; omit -l for global install)
+pi install git:github.com/ethan-blue/pi-charset -l
+
+# Pin a specific release tag
+pi install git:github.com/ethan-blue/pi-charset@v0.1.2 -l
+
+# Or install from a local checkout (run npm install --omit=dev in the package directory first)
+pi install /path/to/pi-charset -l
+```
+
+### 2. Scaffold or Create `.pi-charset.json`
+
+Inside a Pi session, generate `.pi-charset.json` directly with `/charset-init` (auto-scan or preset template):
+
+```text
+/charset-scan                        # Read-only encoding & line-ending profile
+/charset-scan --damaged              # Check for pre-existing U+FFFD / mojibake files
+/charset-init --dry-run              # Preview auto-inferred config without writing disk
+/charset-init                        # Generate .pi-charset.json from project declarations & bytes
+/charset-init embedded-c             # Or scaffold from preset: embedded-c | legacy-java-web | mixed-repo | all-utf8
+/charset-setup                       # Prompt template: guided workspace audit + config preview
+```
+
+Or run the standalone CLI scanner outside Pi:
+
+```bash
+node tools/scan-encoding.mjs /path/to/project --init --dry-run
+node tools/scan-encoding.mjs /path/to/project --init
+```
+
+Or create `.pi-charset.json` manually in the project root (JSONC comments supported):
+
+```jsonc
+{
+  "sourceEncoding": "UTF-8",
+  "writeEncoding": "UTF-8",
+  "unmappable": "error",
+  "verifyWrite": true,
+  "protectUtf8": true,
+  "readStrategy": "auto",
+  "autoCandidates": ["GB18030", "GBK", "GB2312", "Big5"],
+  "overrides": [
+    {
+      "pattern": "src/**/*.c",
+      "encoding": "GBK",
+      "writeEncoding": "GBK",
+      "newFileLineEnding": "CRLF"
+    },
+    {
+      "pattern": "src/**/*.h",
+      "encoding": "GBK",
+      "writeEncoding": "GBK",
+      "newFileLineEnding": "CRLF"
+    }
+  ]
+}
+```
+
+### 3. Diagnostics
+
+Run inside a Pi session:
+
+```text
+/charset-status
+```
+
+## Scope & Boundaries
+
+| Entry Point | Handled | Notes |
+|---|---|---|
+| `read`, `write`, `edit`, `grep` tools | Yes | Works out-of-the-box in zero-config adaptive mode; customizable via optional `.pi-charset.json` |
+| `bash`, `powershell` stdout/stderr | Optional | Disabled by default; enable via `transcodeBash: "auto"` |
+| Shell output redirection (`echo >`, `Set-Content`) | No | Bypasses file tools; use `write` or `edit` instead |
+| CLI startup `@file` arguments | No | Read by the host before extensions load |
+
+Atomic replacement avoids partially written files, but it is not a cross-process transaction or lock. Avoid editing the same file concurrently with external tools. Writes to symbolic links are rejected; edit the target file directly. Existing permission modes are retained, but ownership, ACLs, extended attributes, hard-link relationships, and crash durability are not guaranteed by this replacement scheme.
+
+## Documentation
+
+- [Installation Guide](./docs/INSTALL.md)
+- [Configuration Guide](./docs/CONFIG-GUIDE.md)
+- [Host Compatibility Matrix](./docs/COMPATIBILITY.md)
+- [Acceptance & Test Matrix](./docs/ACCEPTANCE.md)
+- [Release Guide](./docs/RELEASING.md)
+- [Contributing](./CONTRIBUTING.md)
+- [Changelog](./CHANGELOG.md)
+- [Configuration Examples](./examples)
+
+## Development
+
+```bash
+npm run typecheck       # TypeScript type checking (tsc --noEmit)
+npm test                # Run Vitest suite
+npm run check:release   # Verify version sync, changelog, links, and package manifest
+npm run verify          # Run typecheck + test + check:release
+```
+
+## License
+
+[MIT](./LICENSE) © 2026 ethan-blue

package/README.zh-CN.md

@@ -0,0 +1,163 @@
+# pi-charset
+
+[![CI](https://github.com/ethan-blue/pi-charset/actions/workflows/ci.yml/badge.svg)](https://github.com/ethan-blue/pi-charset/actions/workflows/ci.yml)
+[![Version](https://img.shields.io/badge/version-0.1.2-blue.svg)](https://github.com/ethan-blue/pi-charset/releases)
+[![Node](https://img.shields.io/badge/node-%3E%3D22.19.0-brightgreen.svg)](./package.json)
+[![License: MIT](https://img.shields.io/badge/license-MIT-yellow.svg)](./LICENSE)
+
+适用于 [Pi](https://github.com/earendil-works)（`@earendil-works/pi-coding-agent`）的文件编码与行尾处理扩展，在工具 I/O 层接管 `read`、`write`、`edit` 与 `grep`。
+
+[English](./README.md) | 简体中文
+
+---
+
+## 简介
+
+`pi-charset` 在 Pi 的工具 I/O 层接管 `read`、`write`、`edit` 与 `grep`，用于支持同一工程内混合存在 `GBK`、`GB18030`、`Big5`、`Shift_JIS`、`EUC-KR`、`ISO-8859-1`、`UTF-16` 与 `UTF-8` 文件的按文件读写与多编码搜索。全链路在进程内基于字节特征判定，不调用 Python 或外部探测子进程。
+
+## GBK 工程使用建议
+
+已有 GBK 工程，建议先在 Pi 中运行 `/charset-init --dry-run`（或嵌入式 C 工程运行 `/charset-init embedded-c --dry-run`），确认建议的目录规则后再生成 `.pi-charset.json`。保留 `unmappable: "error"` 和 `verifyWrite: true`：GBK 放不下的字符应该报错，而不是静默变成问号。
+
+修改源码时使用 Pi 的 `read`、`edit`、`write` 工具。Shell 重定向和外部格式化工具不经过本扩展。安装后先用 `/charset-status` 查看状态，试改一个已纳入版本管理的小文件，再用常用编辑器核对差异。
+
+字节判定不能证明文件的原始编码。GBK 与 GB18030 存在重叠，同一串字节也可能符合其他编码；工程明确要求 GBK 时，应显式配置 GBK，尤其是新建文件和纯 ASCII 文件。本扩展用于防止新增乱码，不能恢复已经保存为问号或错误文本的原始内容。
+
+## 工作机制
+
+### 1. 字节判定顺序
+
+按以下固定顺序在内存中判定文件编码：
+1. **BOM 识别**：`UTF-8 BOM`、`UTF-16LE`、`UTF-16BE`（`UTF-32 BOM` 归为 `BINARY`）。
+2. **二进制识别**：前 8192 字节内含 `0x00` 判为 `BINARY`。
+3. **显式 `force`**：命中带 `force: true` 的规则时直接采用规则指定编码。
+4. **纯 ASCII**：通过 Node 原生 `node:buffer` `isAscii` 判定全部字节 `<= 0x7F`。
+5. **严格 UTF-8**：通过 Node 原生 `node:buffer` `isUtf8` 整文件零分配校验判为 `UTF-8`。
+6. **多字节候选回环**：按 `autoCandidates` 顺序检查 `encode(decode(bytes)) == bytes` 且无 `U+FFFD`；多候选同时通过时优先采用 `sourceEncoding`。
+7. **兜底**：未通过上述步骤的字节流判为 `UNKNOWN`。
+
+### 2. 三道写入门禁
+
+- **闸门 1（字符可映射检查）**：写入前逐码点检查目标编码表示能力。默认策略 `error` 直接拒写并列出字符及 `U+XXXX` 码点；可选 `escape`（转为 `\uXXXX`）与 `drop-to-gb18030`（GB 族自动升为 `GB18030`）。
+- **闸门 2（写前往返与写后回读校验）**：落盘前在内存验证 `decode(encode(text)) === text`；落盘后回读磁盘文件核对字节与解码文本。若写后核对不一致，报错并保留磁盘现状，不执行旧备份回滚。
+- **闸门 3（未知编码与有损写入拦截）**：拦截 `BINARY` 与 `UNKNOWN` 文件写入；含非 ASCII 的 UTF-8 文件默认保持 UTF-8 写回（`protectUtf8: true`）；磁盘原字节无法在目标编码下逐字节无损回环时拒绝写入。
+
+### 3. 行尾与读取基线校验
+
+- **行尾保持**：已存在文件按磁盘原有行尾（`CRLF` 或 `LF`，UTF-16 按解码后字符级识别）写回；无换行文件不追加尾部换行。
+- **混合换行拦截**：同时含 `CRLF` 与 `LF` 的文件允许 `read` 读取，拒绝 `edit` 与 `write` 修改。
+- **新建文件行尾**：支持通过 `newFileLineEnding: "CRLF" | "LF"` 指定新建文件的行尾格式。
+- **读取基线冲突检测**：读取文件时记录 SHA-1 指纹；写入前若磁盘内容被外部修改或删除，报冲突并拒绝覆盖。
+
+### 4. 零配置自适应与同目录邻居推断
+
+`pi-charset` 安装后无需编写 `.pi-charset.json` 即可直接工作：
+- **已存在文件**：读取时按文件自身字节判定出的编码（`GBK`、`GB18030`、`Big5`、`UTF-16`、`UTF-8`）解码；写入时自动保持原文件编码、原有行尾（`CRLF` 或 `LF`）与 BOM。
+- **新建文件与纯 ASCII 文件加中文**：未放置 `.pi-charset.json` 时，自动扫描同目录下的相邻源码文件（优先匹配同语言后缀族，如 `.c`/`.h`），继承该目录的主流编码与行尾格式（同目录为 UTF-8 或空目录时默认 `UTF-8`）。
+- **多编码搜索（`grep`）**：正则搜索或非 ASCII 字面量搜索时，零配置下执行 `UTF-8 + GB18030` 双趟 `ripgrep` 搜索（有 `.pi-charset.json` 时按配置候选编码集合分趟），经逐文件字节判定复核后按 `file:line` 去重。
+
+## 快速开始
+
+### 1. 安装
+
+```bash
+# 从 Git 安装（-l 写入当前项目 .pi/settings.json，不加 -l 写入用户全局配置）
+pi install git:github.com/ethan-blue/pi-charset -l
+
+# 指定版本标签安装
+pi install git:github.com/ethan-blue/pi-charset@v0.1.2 -l
+
+# 或从本地目录安装（需先在扩展目录执行 npm install --omit=dev）
+pi install /path/to/pi-charset -l
+```
+
+### 2. 脚手架生成或手动编写 `.pi-charset.json`
+
+在 Pi 会话中可直接通过内置脚手架命令扫描或生成 `.pi-charset.json`（无需查找扩展安装路径，落盘后立即刷新当前会话缓存）：
+
+```text
+/charset-scan                        # 只读扫描当前工程的编码与行尾分布
+/charset-scan --damaged              # 排查历史遗留的 U+FFFD / 锟斤拷 受损文件
+/charset-init --dry-run              # 预览自动推断的配置（不写盘）
+/charset-init                        # 根据工程声明与字节特征生成 .pi-charset.json
+/charset-init embedded-c             # 或直接套用预设模板：embedded-c | legacy-java-web | mixed-repo | all-utf8
+/charset-setup                       # 引导式提示模板：扫描审计 + 预览配置
+```
+
+也可在命令行中直接调用独立扫描脚本：
+
+```bash
+node tools/scan-encoding.mjs /path/to/project --init --dry-run
+node tools/scan-encoding.mjs /path/to/project --init
+```
+
+或在项目根目录手动创建 `.pi-charset.json`（支持 `//` 与 `/* */` 注释）：
+
+```jsonc
+{
+  "sourceEncoding": "UTF-8",
+  "writeEncoding": "UTF-8",
+  "unmappable": "error",
+  "verifyWrite": true,
+  "protectUtf8": true,
+  "readStrategy": "auto",
+  "autoCandidates": ["GB18030", "GBK", "GB2312", "Big5"],
+  "overrides": [
+    {
+      "pattern": "src/**/*.c",
+      "encoding": "GBK",
+      "writeEncoding": "GBK",
+      "newFileLineEnding": "CRLF"
+    },
+    {
+      "pattern": "src/**/*.h",
+      "encoding": "GBK",
+      "writeEncoding": "GBK",
+      "newFileLineEnding": "CRLF"
+    }
+  ]
+}
+```
+
+### 3. 状态诊断
+
+在 Pi 会话中执行：
+
+```text
+/charset-status
+```
+
+## 作用边界
+
+| 入口 | 是否处理 | 说明 |
+|---|---|---|
+| `read`、`write`、`edit`、`grep` 工具 | 是 | 默认启用零配置自适应模式；可通过可选的 `.pi-charset.json` 自定义规则 |
+| `bash`、`powershell` 标准输出与错误流 | 可选 | 默认关闭；设置 `transcodeBash: "auto"` 后将非 UTF-8 输出转为 UTF-8 |
+| Shell 重定向写文件（如 `echo >`、`Set-Content`） | 否 | 不经过文件工具层，请使用 `write` 或 `edit` 工具写文件 |
+| 启动命令行 `@file` 参数 | 否 | 在扩展加载前由宿主直接读取 |
+
+原子替换用于避免文件只写了一半，不等于跨进程事务或文件锁。请避免与外部工具同时修改同一文件。符号链接拒绝写入，请直接修改目标文件。替换时保留已有权限模式，但不保证保留所有者、ACL、扩展属性、硬链接关系，也不保证掉电后的持久性。
+
+## 文档索引
+
+- [安装说明](./docs/zh-CN/INSTALL.md)
+- [配置指南](./docs/zh-CN/CONFIG-GUIDE.md)
+- [宿主兼容矩阵](./docs/zh-CN/COMPATIBILITY.md)
+- [验收与测试矩阵](./docs/zh-CN/ACCEPTANCE.md)
+- [发布规范](./docs/zh-CN/RELEASING.md)
+- [参与贡献](./CONTRIBUTING.zh-CN.md)
+- [版本变更日志](./CHANGELOG.zh-CN.md)
+- [配置示例](./examples)
+
+## 本地开发
+
+```bash
+npm run typecheck       # TypeScript 类型检查 (tsc --noEmit)
+npm test                # 运行 Vitest 测试套件
+npm run check:release   # 校验版本一致性、变更日志、文档链接与打包清单
+npm run verify          # 依次运行 typecheck + test + check:release
+```
+
+## 许可证
+
+[MIT](./LICENSE) © 2026 ethan-blue

package/docs/ACCEPTANCE.md

@@ -0,0 +1,70 @@
+# Acceptance & Test Matrix
+
+English | [简体中文](./zh-CN/ACCEPTANCE.md)
+
+---
+
+## Verification Commands
+
+```bash
+npm run typecheck       # tsc --noEmit (0 errors)
+npm test                # vitest run; reports current pass/skip counts
+npm run check:release   # version sync, changelog, markdown links, and tarball manifest
+npm run verify          # typecheck + test + check:release
+```
+
+The matrix below maps expected behavior to test suites; it is not a live CI report. Use the commands above to verify the current checkout. Check skipped tests as well as failures: host integration and platform-specific cases depend on the installed host and operating system. A green Windows run does not establish POSIX permission or symlink behavior, and a smoke performance threshold is not a latency guarantee.
+
+## 1. End-to-End Acceptance Matrix (A-01 ~ A-24)
+
+| ID | Scenario | Status | Verification Suite |
+|---|---|---|---|
+| A-01 | Read GBK Chinese source files | PASS | `test/acceptance.test.ts` (with full-tree SHA-256 verification) |
+| A-02 | Edit GBK Chinese and numeric literals preserving encoding and `CRLF` | PASS | `test/acceptance.test.ts`, `test/host-integration.test.ts` |
+| A-03 | Zero-config self-adaptive read/write/sibling inference without `.pi-charset.json` | PASS | `test/zero-config.test.ts`, `test/operations.test.ts` |
+| A-04 | Preserve single BOM when editing UTF-8 BOM files | PASS | `test/acceptance.test.ts`, `test/roundtrip.test.ts` |
+| A-05 | Write pure-ASCII files as GBK when Chinese characters are first added | PASS | `test/acceptance.test.ts`, `test/roundtrip.test.ts` |
+| A-06 | Apply `newFileLineEnding` (`GBK + CRLF`) to new `.c`/`.h` files; write new docs as `UTF-8` | PASS | `test/acceptance.test.ts`, `test/operations.test.ts` |
+| A-07 | Preserve absence of trailing newline on edit | PASS | `test/acceptance.test.ts`, `test/operations.test.ts` |
+| A-08 | Allow reading `MIXED` (`CRLF` + `LF`) files while rejecting `edit` and `write` | PASS | `test/acceptance.test.ts`, `test/operations.test.ts`, `test/regressions.test.ts` |
+| A-09 | Reject unmappable code points (Gate 1) without writing `0x3F` (`?`) | PASS | `test/gates.test.ts`, `test/acceptance.test.ts` |
+| A-10 | Classify truncated or invalid byte sequences as `UNKNOWN` and reject writes | PASS | `test/classify.test.ts`, `test/resolve.test.ts`, `test/gates.test.ts` |
+| A-11 | Return raw image bytes and `image` content block for PNG/JPEG/GIF/WebP/BMP | PASS | `test/operations.test.ts`, `test/acceptance.test.ts` |
+| A-12 | Search Chinese keywords across mixed GBK and UTF-8 files without duplicate hits | PASS | `test/grep.test.ts`, `test/acceptance.test.ts`, `test/perf/grep-scale.test.ts` |
+| A-13 | Support `glob`, `literal`, `ignoreCase`, `context`, `limit`, and `AbortSignal` in `grep` | PASS | `test/grep.test.ts`, `test/perf/grep-scale.test.ts` |
+| A-14 | Exclude `.git`, `.svn`, `.hg`, and `node_modules` from search results | PASS | `test/grep.test.ts`, `test/acceptance.test.ts` |
+| A-15 | Emit warning and apply safe defaults when `.pi-charset.json` contains malformed JSON | PASS | `test/acceptance.test.ts`, `test/resolve.test.ts` |
+| A-16 | Detect external file modification or deletion via SHA-1 read baseline and reject stale writes | PASS | `test/acceptance.test.ts`, `test/operations.test.ts`, `test/regressions.test.ts` |
+| A-17 | Preserve on-disk file state without rollback when post-write verification detects conflict | PASS | `test/gates.test.ts`, `test/regressions.test.ts` |
+| A-18 | Report write errors on read-only or locked files and clean up temporary `.tmp` files | PASS | `test/acceptance.test.ts`, `test/gates.test.ts` |
+| A-19 | Serialize 20 concurrent writes to the same file without corruption or leftover temp files | PASS | `test/concurrency.test.ts` |
+| A-20 | Invalidate caches automatically on config modification or session reload | PASS | `test/resolve.test.ts`, `test/edit-preview.test.ts` |
+| A-21 | Sub-session tool inheritance | N/A | Host-dependent; documented in [`README.md`](../README.md) scope table |
+| A-22 | Load `src/index.ts` via `discoverAndLoadExtensions` and CLI RPC mode | PASS | `test/host-integration.test.ts` |
+| A-23 | Compute encoding-aware `edit` preview diffs and invalidate preview cache | PASS | `test/edit-preview.test.ts` |
+| A-24 | Verify `npm pack --dry-run` file whitelist and clean directory loading | PASS | `tools/check-release.mjs` |
+
+## 2. Test Suite Breakdown
+
+| Test File | Coverage |
+|---|---|
+| `test/classify.test.ts` | 16 fixture classifications, UTF-8 priority, single-byte exclusion, 1MB classification latency & cache |
+| `test/converter.test.ts` | Encoding alias normalization, GB-family detection, unmappable code point detection & `\uXXXX` escaping |
+| `test/regressions.test.ts` | GBK question marks and emoji; UTF-8/UTF-16 line endings and BOM; external deletion and pre-rename conflicts; platform path keys; unsupported shell encodings; Rust regex; POSIX mode/symlink checks (platform-conditional) |
+| `test/config.test.ts` | `.pi-charset.json` nearest-directory lookup, override specificity scoring & validation |
+| `test/resolve.test.ts` | Read/write decision matrix, `writeIntent` resolution, UTF-8 protection, BOM rules, config hot-reload |
+| `test/gates.test.ts` | Gate 1 (`error`/`escape`/`drop-to-gb18030`), Gate 2 (read-back verification preserving disk state), Gate 3 |
+| `test/roundtrip.test.ts` | Byte-exact `read -> write` round-trip across 12 encodings, ASCII-to-GBK writes, `CRLF` idempotency |
+| `test/utf8-protect.test.ts` | UTF-8 preservation inside GBK trees and full-tree mixed-encoding verification |
+| `test/line-endings.test.ts` | `CRLF`, `LF`, `MIXED`, and `NONE` detection and restoration |
+| `test/operations.test.ts` | `read`/`write`/`edit` operations, `newFileLineEnding`, mixed line-ending guard, SHA-1 baseline check |
+| `test/grep.test.ts` | Multi-pass search, per-file classification filtering, context decoding, zero-config multi-encoding search |
+| `test/edit-preview.test.ts` | Synchronous preview routing cache, async diff rendering for GB files, single-byte `force` routing |
+| `test/shell.test.ts` | Streaming shell output transcoding, split multi-byte chunks, incomplete UTF-8 tail detection, file dump detection |
+| `test/sysnote.test.ts` | Conditional system prompt injection and `tool_result` notice queue |
+| `test/zero-config.test.ts` | Zero-config self-adaptive read, write, edit, and same-directory sibling inference |
+| `test/scan-init.test.ts` | `tools/scan-encoding.mjs` CLI validation, `--init` declaration extraction, narrowest encoding selection, `--damaged` |
+| `test/concurrency.test.ts` | 20-way concurrent single-file writes and multi-file concurrent writes |
+| `test/host-integration.test.ts` | Extension registration, `/charset-status` + `/charset-init` + `/charset-scan` scaffolding commands, and `discoverAndLoadExtensions` loader test |
+| `test/perf/io-latency.test.ts` | 300x 8.7KB GBK read/write loop and 1MB file read/write latency |
+| `test/perf/grep-scale.test.ts` | 2,000-file mixed GBK/UTF-8 search and `AbortSignal` cancellation latency |

package/docs/COMPATIBILITY.md

@@ -0,0 +1,63 @@
+# Host Compatibility & Verification
+
+English | [简体中文](./zh-CN/COMPATIBILITY.md)
+
+---
+
+## 1. Tested Host Matrix
+
+`package.json` declares `peerDependencies` for `@earendil-works/pi-coding-agent >= 0.84.0`. The table below lists verified host environments:
+
+| Environment | Host Version | Typecheck | Full Test Suite | Extension Loader (`discoverAndLoadExtensions`) | CLI Smoke (`pi --mode rpc`) |
+|---|---|---|---|---|---|
+| v0.1.2 lockfile, Windows / Node 24 | `1.4.0` | PASS | 271 passed, 2 POSIX-only tests skipped | PASS | — |
+| v0.1.1 lockfile, Windows / Node 24 | `1.4.0` | PASS | 270 passed, 2 POSIX-only tests skipped | PASS | — |
+| Historical v0.1.0 lockfile | `0.84.3` | PASS | PASS | PASS | — |
+| Local Host Junction | `1.2.0` / `1.2.0` / `1.2.0` / `1.2.0` | PASS | PASS | PASS | PASS |
+| Clean Tarball Unpack | `1.4.0` | — | — | PASS | — |
+
+See GitHub Actions for current Windows/Linux and Node 22/24 results.
+
+The root `overrides` entry keeps the checkout's host dependency on a patched `undici`. npm ignores dependency-package overrides when this extension is installed into another application: update and audit your actual Pi installation separately.
+
+## 2. Cross-Version API Adaptations (`0.84.3` – `1.4.0`)
+
+| Location | Host Change | Adaptation |
+|---|---|---|
+| `src/grep.ts` tool `execute` 5th parameter | `ExtensionContext` in `0.84.3` -> `ExtensionToolContext` in `1.2.0+` (adds `tools` and `executeTool`) | Derived via `Parameters<typeof builtin.execute>[4]` so all versions typecheck cleanly |
+| `test/host-integration.test.ts` | Extension loader verification | Calls `discoverAndLoadExtensions` exported by `@earendil-works/pi-coding-agent` |
+| `src/index.ts` command registration | `pi.registerCommand` | Registers `/charset-status`, `/charset-init`, and `/charset-scan`, verified across host versions |
+
+## 3. Host Junction Verification Workflow (Windows)
+
+To run the full test suite against a globally installed Pi version without modifying the repository's lockfile:
+
+```powershell
+# 1. Locate global npm root and installed Pi version
+npm root -g
+pi --version
+
+# 2. Copy repository excluding node_modules and .git
+robocopy D:\path\to\pi-charset D:\tmp\pi-host-check /E /XD node_modules .git
+
+# 3. Junction non-@earendil-works dependencies from the repository lockfile
+node -e "const fs=require('fs'),p=require('path');const src='D:/path/to/pi-charset/node_modules',dst='D:/tmp/pi-host-check/node_modules';fs.mkdirSync(dst,{recursive:true});for(const e of fs.readdirSync(src)){if(e==='@earendil-works')continue;fs.symlinkSync(p.join(src,e),p.join(dst,e),'junction')}"
+
+# 4. Junction @earendil-works packages from the global Pi installation
+node -e "const fs=require('fs');const g='<npm-root-g>/@earendil-works/pi-coding-agent',dst='D:/tmp/pi-host-check/node_modules/@earendil-works';fs.mkdirSync(dst,{recursive:true});for(const n of ['pi-coding-agent','pi-ai','pi-tui','pi-agent-core'])fs.symlinkSync(n==='pi-coding-agent'?g:g+'/node_modules/@earendil-works/'+n,dst+'/'+n,'junction')"
+
+# 5. Run typecheck and full test suite (PI_HOST_VERSION asserts the active host version)
+cd D:\tmp\pi-host-check
+npm run typecheck
+$env:PI_HOST_VERSION='1.2.0'; npx vitest run
+
+# 6. CLI RPC smoke check
+cd $env:TEMP
+pi --mode rpc -e D:/tmp/pi-host-check/src/index.ts --no-mcp --no-session < NUL
+```
+
+## 4. Version Consistency Assertions
+
+`test/host-integration.test.ts` enforces two version invariants:
+1. The runtime `VERSION` exported by `@earendil-works/pi-coding-agent` must match `node_modules/@earendil-works/pi-coding-agent/package.json`.
+2. When `PI_HOST_VERSION` is set in the environment, `VERSION` must equal `process.env.PI_HOST_VERSION`.