docs-i18n 0.8.0 → 0.8.2

package/template/content/en/architecture.md

@@ -0,0 +1,222 @@
+---
+title: Architecture
+description: How docs-i18n works internally -- the translation pipeline, AST parsing, caching, and chunking strategies.
+---
+
+# Architecture
+
+This document explains how docs-i18n works internally. Understanding the pipeline helps you tune configuration, debug issues, and contribute to the project.
+
+## Translation Pipeline
+
+The end-to-end flow is:
+
+```
+Source files (EN)
+    |
+    v
+[1. Normalize] -- Ensure JSX tags are separated by blank lines
+    |
+    v
+[2. Parse] -- remark AST -> flat list of typed nodes
+    |
+    v
+[3. Hash] -- MD5 of each translatable node's text
+    |
+    v
+[4. Chunk] -- Group nodes into chunks that fit the LLM context window
+    |
+    v
+[5. Translate] -- Send JSON to LLM, receive JSON translations
+    |
+    v
+[6. Cache] -- Store translations in SQLite keyed by (lang, md5)
+    |
+    v
+[7. Assemble] -- EN source + cache -> translated output files
+```
+
+## Step 1: Normalization
+
+The `normalize()` function (`src/core/normalize.ts`) preprocesses MDX content to ensure that JSX tags like `<AppOnly>`, `<PagesOnly>`, `<details>`, and `<div>` are separated from surrounding content by blank lines. This ensures remark parses them as independent HTML nodes rather than merging them with adjacent text.
+
+For example, this input:
+
+```
+<AppOnly>
+Some text here
+</AppOnly>
+```
+
+Becomes:
+
+```
+<AppOnly>
+
+Some text here
+
+</AppOnly>
+```
+
+## Step 2: AST Parsing
+
+The `parseMdx()` function (`src/core/parser.ts`) uses remark to parse markdown into a flat list of `ParsedNode` objects. Each node has:
+
+- `type` -- The AST node type: `paragraph`, `heading`, `list`, `blockquote`, `code`, `html`, `thematicBreak`, or `frontmatter`.
+- `rawText` -- The raw text content from the normalized source.
+- `needsTranslation` -- Whether this node contains human-readable text.
+- `md5` -- MD5 hash of the raw text (only for translatable nodes).
+- `startOffset` / `endOffset` -- Character offsets in the normalized content.
+
+**Translatable node types:** `paragraph`, `heading`, `list`, `blockquote`, and `html` nodes that contain non-tag text (e.g., `<summary>Examples</summary>`).
+
+**Non-translatable:** `code` blocks, `thematicBreak`, and pure HTML/JSX tags (self-closing tags like `<Check size={18} />`, opening/closing tags like `<AppOnly></AppOnly>`).
+
+**Frontmatter handling:** If the content starts with `---`, the parser detects YAML frontmatter and emits it as a single `frontmatter` node spanning from the opening `---` to the closing `---`. The frontmatter module (`src/core/frontmatter.ts`) then extracts only the configured translatable fields (e.g., `title`, `description`) using the `yaml` library, sends them to the LLM as plain text, and reconstructs the YAML with translated values while preserving all other fields and formatting.
+
+## Step 3: MD5 Hashing
+
+Each translatable node's raw text is hashed with MD5 to produce a stable key. This key is used for:
+
+- **Deduplication** -- identical content appearing in multiple files (or even multiple projects) shares a single translation.
+- **Incremental updates** -- when source content changes, only the nodes with new MD5 hashes need translation. Unchanged nodes reuse their cached translations.
+- **Heading differentiation** -- heading nodes include their level markers (`##`, `###`) in the hash, so "## Installation" and "### Installation" produce different keys.
+
+## Step 4: Smart Chunking
+
+The translate command groups untranslated nodes into chunks that fit within the LLM's context window. The chunking algorithm (`src/commands/translate.ts`) accounts for:
+
+- **Input budget** -- system prompt tokens + source text tokens. Estimated at `text.length / 4 + 80` tokens per node (accounting for JSON structure overhead).
+- **Output budget** -- translated text tokens. Scaled by a per-language multiplier since different languages use tokens differently:
+
+| Language | Multiplier | Reason |
+| --- | --- | --- |
+| Japanese, Korean, Hindi, Thai | 2.5x | CJK/Indic tokenization |
+| Russian, Arabic, Ukrainian, Hebrew | 2.0x | Cyrillic/Arabic scripts |
+| Chinese (Simplified/Traditional) | 1.5x | CJK but more concise |
+| German, French, Portuguese | 1.3x | Slightly longer than English |
+| Spanish, Vietnamese | 1.2x | Close to English length |
+| Other languages | 2.0x | Safe default |
+
+- **System prompt overhead** -- approximately 700 tokens for the translation prompt.
+- **JSON structure overhead** -- approximately 80 tokens per key for JSON schema properties.
+- **Safety margin** -- 85% of input budget and 75% of output budget are used to leave room for estimation errors.
+
+When a chunk reaches its budget, a new chunk is started. This prevents context window overflow and output truncation.
+
+## Step 5: LLM Translation
+
+The translator (`src/core/translator.ts`) uses structured JSON mode for translations:
+
+**Input format:** A JSON object with a `nodes` array. Each node has a `key` (MD5), `type` (heading/paragraph/list/etc.), and `text` (content to translate).
+
+```json
+{
+  "nodes": [
+    { "key": "a1b2c3...", "type": "heading", "text": "## Installation" },
+    { "key": "d4e5f6...", "type": "paragraph", "text": "Run the following command:" }
+  ]
+}
+```
+
+**Output format:** A flat JSON object mapping each key to its translation.
+
+```json
+{
+  "a1b2c3...": "## \u5b89\u88c5",
+  "d4e5f6...": "\u8fd0\u884c\u4ee5\u4e0b\u547d\u4ee4\uff1a"
+}
+```
+
+**Robustness features:**
+
+- **JSON repair** -- Handles common LLM JSON errors: unescaped newlines in strings, trailing commas, missing closing braces.
+- **Thinking block stripping** -- Removes `<think>...</think>` blocks from reasoning models.
+- **Unwrapping** -- If the model wraps output in `{"nodes": {...}}` or `{"translations": {...}}`, it is automatically unwrapped.
+- **Key recovery** -- If the model corrupts an MD5 key (e.g., truncates it), the translator attempts fuzzy matching to recover the translation (up to 3 character differences).
+- **Garbage detection** -- If more than 50% of translation values are identical, the model output is rejected.
+- **Retry with backoff** -- Retries on 429 (rate limit), 503, 405, timeout, and connection errors. Uses exponential backoff starting at 2 seconds.
+- **Model rotation** -- Supports a list of models to rotate through. Dead models (400/404 errors) are skipped. Rate-limited models (429) are deprioritized.
+- **Truncation detection** -- If `finish_reason` is `'length'`, the output was truncated and the request is retried.
+
+**Frontmatter translation:**
+
+Frontmatter nodes are handled specially. Instead of sending the entire YAML block, the translator extracts individual translatable fields (e.g., `title`, `description`) and sends them as plain `paragraph` type nodes with virtual keys like `fm:<md5>:title`. After translation, the fields are reassembled into the original YAML structure using `reconstructFrontmatter()`.
+
+## Step 6: SQLite Cache
+
+The `TranslationCache` class (`src/core/cache.ts`) manages all persistent state in a single SQLite database at `<cacheDir>/translations.db`.
+
+**Schema:**
+
+```sql
+-- EN source texts (deduplicated by MD5)
+CREATE TABLE sources (
+  key   TEXT PRIMARY KEY NOT NULL,  -- MD5 hash
+  text  TEXT NOT NULL,              -- original English text
+  type  TEXT NOT NULL DEFAULT 'paragraph'
+);
+
+-- Which files use each source node
+CREATE TABLE source_files (
+  key     TEXT    NOT NULL,  -- MD5 hash
+  file    TEXT    NOT NULL,  -- relative file path
+  line    INTEGER NOT NULL,  -- line number
+  version TEXT    NOT NULL DEFAULT 'latest',
+  PRIMARY KEY (version, key, file, line)
+);
+
+-- Translated texts per language
+CREATE TABLE translations (
+  lang       TEXT    NOT NULL,  -- language code
+  key        TEXT    NOT NULL,  -- MD5 hash
+  value      TEXT    NOT NULL,  -- translated text
+  created_at INTEGER NOT NULL DEFAULT (unixepoch()),
+  updated_at INTEGER NOT NULL DEFAULT (unixepoch()),
+  PRIMARY KEY (lang, key)
+);
+```
+
+**Performance configuration:**
+
+- **WAL mode** -- Write-Ahead Logging enables concurrent readers with a single writer. No locking issues during parallel translation and assembly.
+- **busy_timeout = 5000** -- Wait up to 5 seconds for a lock before failing.
+- **synchronous = NORMAL** -- Balanced between safety and performance.
+- **WITHOUT ROWID** -- Tables use the primary key directly, avoiding an extra rowid column.
+- **Immediate writes** -- All `set()` calls write directly to disk. No explicit save step needed.
+
+**Key operations:**
+
+- `get(lang, md5)` -- Look up a cached translation.
+- `set(lang, md5, translation)` -- Store or update a translation (upsert).
+- `untranslatedKeys(lang, version)` -- Find all source keys that have no translation for a given language.
+- `fileCoverage(version, lang)` -- Get per-file translation coverage (used by the admin dashboard).
+- `prune(lang, usedMd5s)` -- Remove translations whose keys are no longer referenced.
+- `exportJsonl(lang, outputPath)` / `importJsonl(lang, inputPath)` -- Export/import translations in JSONL format for backup or migration.
+
+**SQLite compatibility:**
+
+The `openDatabase()` function (`src/core/sqlite.ts`) automatically detects the runtime environment. Under Bun, it uses `bun:sqlite`. Under Node.js, it uses `better-sqlite3`. Both expose the same interface.
+
+## Step 7: Assembly
+
+The `assemble()` function (`src/core/assembler.ts`) produces a translated file from English source content and cached translations:
+
+1. Normalizes the source content.
+2. Parses it into AST nodes.
+3. For each node:
+   - **Non-translatable nodes** (code blocks, HTML tags) are kept as-is.
+   - **Translatable nodes with a cached translation** are replaced with the cached value.
+   - **Translatable nodes without a cache hit** are either wrapped in `<!-- NEEDS_TRANSLATION -->` markers (for the legacy whole-file translation mode) or fall back to the original English text (for assembled output files).
+4. Preserves all whitespace and newlines between nodes.
+
+The `AssembleResult` includes statistics: `cachedCount`, `uncachedCount`, `totalTranslatable`, and whether all nodes were cached (`allCached`).
+
+## Validation
+
+The `validate()` function (`src/core/validator.ts`) compares LLM output against the translation cache to detect and correct modifications to already-cached translations. It uses two alignment strategies:
+
+- **Fast path** -- When the number of translatable nodes in the source and output match, nodes are aligned by index.
+- **Anchor-based alignment** -- When node counts differ (the LLM merged or split paragraphs), cached translations serve as anchor points. The validator finds exact matches between output text and cached translations, then aligns nodes between anchors by type matching.
+
+Cached translations always override LLM modifications, ensuring translation consistency across runs.

package/template/content/en/cli.md

@@ -0,0 +1,269 @@
+---
+title: CLI Reference
+description: Complete reference for all docs-i18n CLI commands, flags, and common workflows.
+---
+
+# CLI Reference
+
+docs-i18n provides a command-line interface for managing documentation translations. All commands read from `docs-i18n.config.ts` in the current working directory by default.
+
+```bash
+docs-i18n <command> [options]
+```
+
+## Global Options
+
+| Flag | Description |
+| --- | --- |
+| `--config <path>` | Path to config file (default: `docs-i18n.config.ts`) |
+| `--version`, `-v` | Print the docs-i18n version |
+| `--help`, `-h` | Show help text |
+
+## Commands
+
+### `translate`
+
+Translate content to a target language. Sends untranslated nodes to the configured LLM and caches the results.
+
+```bash
+docs-i18n translate --lang <code> [options]
+```
+
+**Required:**
+
+| Flag | Description |
+| --- | --- |
+| `--lang <code>` | Target language code (e.g., `zh-hans`, `ja`, `es`) |
+
+**Optional:**
+
+| Flag | Default | Description |
+| --- | --- | --- |
+| `--project <id>` | all projects | Filter to a specific project |
+| `--version <ver>` | all versions | Filter to a specific version |
+| `--files <paths>` | all files | Comma-separated list of relative file paths to translate |
+| `--max <n>` | 999 | Maximum number of API call chunks to process |
+| `--concurrency <n>` | 3 | Number of parallel API calls |
+| `--model <model>` | config value | Override the LLM model |
+| `--api-key <key>` | config/env | Override the API key |
+| `--max-tokens <n>` | 16384 | Max output tokens per API call |
+| `--context-length <n>` | 32768 | Model context window size |
+| `--dry-run` | false | Preview what would be translated without making API calls |
+
+**Examples:**
+
+```bash
+# Translate everything to Simplified Chinese
+docs-i18n translate --lang zh-hans
+
+# Translate a specific version with dry run
+docs-i18n translate --lang zh-hans --version latest --dry-run
+
+# Translate specific files only
+docs-i18n translate --lang zh-hans --files docs/intro.mdx,docs/guide.mdx
+
+# Use a different model with higher token limits
+docs-i18n translate --lang ja --model qwen/qwen3.5-flash-02-23 --max-tokens 65536 --context-length 1000000
+
+# Limit API calls for testing
+docs-i18n translate --lang es --max 5 --concurrency 1
+```
+
+**How it works:**
+
+1. Loads the SQLite cache and finds all untranslated keys for the given language and version.
+2. Groups untranslated nodes into chunks that fit within the model's context window, accounting for input tokens, output tokens, and language-specific token multipliers (e.g., Japanese uses 2.5x more output tokens than English).
+3. Sends each chunk to the LLM as structured JSON: an array of typed nodes with MD5 keys.
+4. Parses the JSON response, validates keys, and stores translations in the cache.
+5. Runs chunks in parallel up to the concurrency limit.
+
+Translation logs are written to `.logs/` with timestamps for debugging.
+
+---
+
+### `assemble`
+
+Assemble translated files by combining English source content with cached translations. For any untranslated nodes, the original English text is used as a fallback.
+
+```bash
+docs-i18n assemble [options]
+```
+
+**Optional:**
+
+| Flag | Default | Description |
+| --- | --- | --- |
+| `--project <id>` | all projects | Filter to a specific project |
+| `--version <ver>` | all versions | Filter to a specific version |
+| `--lang <code>` | all languages | Filter to a specific language |
+
+**Examples:**
+
+```bash
+# Assemble all projects, versions, and languages
+docs-i18n assemble
+
+# Assemble only Chinese for the latest version
+docs-i18n assemble --lang zh-hans --version latest
+```
+
+Output is written to `.cache/content/<version>/<lang>/` by default. Each file mirrors the structure of the English source directory.
+
+---
+
+### `rescan`
+
+Rescan source files and rebuild the source index. Also cleans orphaned translations and sources that no longer exist in the source files.
+
+```bash
+docs-i18n rescan [options]
+```
+
+**Optional:**
+
+| Flag | Default | Description |
+| --- | --- | --- |
+| `--project <id>` | all projects | Filter to a specific project |
+| `--version <ver>` | all versions | Filter to a specific version |
+
+**Examples:**
+
+```bash
+# Rescan all source files
+docs-i18n rescan
+
+# Rescan a specific version only
+docs-i18n rescan --version latest
+```
+
+**What it does:**
+
+1. Walks all source directories and parses every markdown/MDX file.
+2. Extracts translatable nodes and stores their MD5 keys, source text, node types, file paths, and line numbers in the SQLite cache.
+3. Deletes orphaned translations whose source keys no longer exist in any source file.
+4. Deletes orphaned source entries that are no longer referenced by any file.
+
+Run `rescan` after adding, removing, or significantly editing source files.
+
+---
+
+### `status`
+
+Show translation coverage for all projects and versions.
+
+```bash
+docs-i18n status [options]
+```
+
+**Optional:**
+
+| Flag | Default | Description |
+| --- | --- | --- |
+| `--lang <code>` | all languages | Show status for a specific language only |
+
+**Examples:**
+
+```bash
+# Show status for all languages
+docs-i18n status
+
+# Show status for Chinese only
+docs-i18n status --lang zh-hans
+```
+
+**Output:**
+
+```
+Translation Status
+
+latest (1523 keys):
+  zh-hans    ████████████████████ 100% (1523/1523)
+  ja         ████████████░░░░░░░░ 62% (944/1523)
+  es         ██░░░░░░░░░░░░░░░░░░ 10% (152/1523)
+```
+
+If a version shows "no source files (run rescan first)", you need to run `docs-i18n rescan` before checking status.
+
+---
+
+### `admin`
+
+Start the admin dashboard web UI for managing translations.
+
+```bash
+docs-i18n admin [options]
+```
+
+**Optional:**
+
+| Flag | Default | Description |
+| --- | --- | --- |
+| `--port <n>` | 3456 | Port to run the dashboard on |
+
+**Examples:**
+
+```bash
+# Start on default port
+docs-i18n admin
+
+# Start on custom port
+docs-i18n admin --port 4000
+```
+
+The dashboard opens at `http://localhost:3456` (or your custom port). See the [Admin Dashboard](./admin.md) documentation for details on its features.
+
+## Common Workflows
+
+### Full translation pipeline
+
+```bash
+# 1. Scan source files
+docs-i18n rescan
+
+# 2. Check what needs translating
+docs-i18n status
+
+# 3. Translate
+docs-i18n translate --lang zh-hans
+
+# 4. Assemble output
+docs-i18n assemble --lang zh-hans
+```
+
+### Translate a single file for review
+
+```bash
+docs-i18n translate --lang zh-hans --files docs/getting-started.mdx --max 1
+docs-i18n assemble --lang zh-hans
+```
+
+### Preview translation volume before spending API credits
+
+```bash
+docs-i18n translate --lang ja --dry-run
+```
+
+This reports the number of untranslated keys and chunks without making any API calls.
+
+### Multi-project translation
+
+```bash
+# Translate only the "query" project
+docs-i18n translate --lang zh-hans --project query
+
+# Assemble only the "table" project
+docs-i18n assemble --project table --lang zh-hans
+```
+
+### After editing source files
+
+```bash
+# Rescan to detect changes and clean orphans
+docs-i18n rescan
+
+# Translate only new/changed content (cached translations are reused)
+docs-i18n translate --lang zh-hans
+
+# Rebuild output files
+docs-i18n assemble
+```