docs-i18n 0.8.0 → 0.8.2

package/template/content/docs-i18n/en/cli.md

@@ -0,0 +1,324 @@
+---
+title: CLI Reference
+description: Complete reference for all docs-i18n CLI commands, flags, and common workflows.
+---
+
+# CLI Reference
+
+docs-i18n provides a command-line interface for managing documentation translations. All commands read from `docs-i18n.config.ts` in the current working directory by default.
+
+```bash
+docs-i18n <command> [options]
+```
+
+## Global Options
+
+| Flag | Description |
+| --- | --- |
+| `--config <path>` | Path to config file (default: `docs-i18n.config.ts`) |
+| `--version`, `-v` | Print the docs-i18n version |
+| `--help`, `-h` | Show help text |
+
+## Commands
+
+### `translate`
+
+Translate content to a target language. Sends untranslated nodes to the configured LLM and caches the results.
+
+```bash
+docs-i18n translate --lang <code> [options]
+```
+
+**Required:**
+
+| Flag | Description |
+| --- | --- |
+| `--lang <code>` | Target language code (e.g., `zh-hans`, `ja`, `es`) |
+
+**Optional:**
+
+| Flag | Default | Description |
+| --- | --- | --- |
+| `--project <id>` | all projects | Filter to a specific project |
+| `--version <ver>` | all versions | Filter to a specific version |
+| `--files <paths>` | all files | Comma-separated list of relative file paths to translate |
+| `--max <n>` | 999 | Maximum number of API call chunks to process |
+| `--concurrency <n>` | 3 | Number of parallel API calls |
+| `--model <model>` | config value | Override the LLM model |
+| `--api-key <key>` | config/env | Override the API key |
+| `--max-tokens <n>` | 16384 | Max output tokens per API call |
+| `--context-length <n>` | 32768 | Model context window size |
+| `--dry-run` | false | Preview what would be translated without making API calls |
+
+**Examples:**
+
+```bash
+# Translate everything to Simplified Chinese
+docs-i18n translate --lang zh-hans
+
+# Translate a specific version with dry run
+docs-i18n translate --lang zh-hans --version latest --dry-run
+
+# Translate specific files only
+docs-i18n translate --lang zh-hans --files docs/intro.mdx,docs/guide.mdx
+
+# Use a different model with higher token limits
+docs-i18n translate --lang ja --model qwen/qwen3.5-flash-02-23 --max-tokens 65536 --context-length 1000000
+
+# Limit API calls for testing
+docs-i18n translate --lang es --max 5 --concurrency 1
+```
+
+**How it works:**
+
+1. Loads the SQLite cache and finds all untranslated keys for the given language and version.
+2. Groups untranslated nodes into chunks that fit within the model's context window, accounting for input tokens, output tokens, and language-specific token multipliers (e.g., Japanese uses 2.5x more output tokens than English).
+3. Sends each chunk to the LLM as structured JSON: an array of typed nodes with MD5 keys.
+4. Parses the JSON response, validates keys, and stores translations in the cache.
+5. Runs chunks in parallel up to the concurrency limit.
+
+Translation logs are written to `.logs/` with timestamps for debugging.
+
+---
+
+### `assemble`
+
+Assemble translated files by combining English source content with cached translations. For any untranslated nodes, the original English text is used as a fallback.
+
+```bash
+docs-i18n assemble [options]
+```
+
+**Optional:**
+
+| Flag | Default | Description |
+| --- | --- | --- |
+| `--project <id>` | all projects | Filter to a specific project |
+| `--version <ver>` | all versions | Filter to a specific version |
+| `--lang <code>` | all languages | Filter to a specific language |
+
+**Examples:**
+
+```bash
+# Assemble all projects, versions, and languages
+docs-i18n assemble
+
+# Assemble only Chinese for the latest version
+docs-i18n assemble --lang zh-hans --version latest
+```
+
+Output is written to `.cache/content/<version>/<lang>/` by default. Each file mirrors the structure of the English source directory.
+
+---
+
+### `rescan`
+
+Rescan source files and rebuild the source index. Also cleans orphaned translations and sources that no longer exist in the source files.
+
+```bash
+docs-i18n rescan [options]
+```
+
+**Optional:**
+
+| Flag | Default | Description |
+| --- | --- | --- |
+| `--project <id>` | all projects | Filter to a specific project |
+| `--version <ver>` | all versions | Filter to a specific version |
+
+**Examples:**
+
+```bash
+# Rescan all source files
+docs-i18n rescan
+
+# Rescan a specific version only
+docs-i18n rescan --version latest
+```
+
+**What it does:**
+
+1. Walks all source directories and parses every markdown/MDX file.
+2. Extracts translatable nodes and stores their MD5 keys, source text, node types, file paths, and line numbers in the SQLite cache.
+3. Deletes orphaned translations whose source keys no longer exist in any source file.
+4. Deletes orphaned source entries that are no longer referenced by any file.
+
+Run `rescan` after adding, removing, or significantly editing source files.
+
+---
+
+### `status`
+
+Show translation coverage for all projects and versions.
+
+```bash
+docs-i18n status [options]
+```
+
+**Optional:**
+
+| Flag | Default | Description |
+| --- | --- | --- |
+| `--lang <code>` | all languages | Show status for a specific language only |
+
+**Examples:**
+
+```bash
+# Show status for all languages
+docs-i18n status
+
+# Show status for Chinese only
+docs-i18n status --lang zh-hans
+```
+
+**Output:**
+
+```
+Translation Status
+
+latest (1523 keys):
+  zh-hans    ████████████████████ 100% (1523/1523)
+  ja         ████████████░░░░░░░░ 62% (944/1523)
+  es         ██░░░░░░░░░░░░░░░░░░ 10% (152/1523)
+```
+
+If a version shows "no source files (run rescan first)", you need to run `docs-i18n rescan` before checking status.
+
+---
+
+### `admin`
+
+Start the admin dashboard web UI for managing translations.
+
+```bash
+docs-i18n admin [options]
+```
+
+**Optional:**
+
+| Flag | Default | Description |
+| --- | --- | --- |
+| `--port <n>` | 3456 | Port to run the dashboard on |
+
+**Examples:**
+
+```bash
+# Start on default port
+docs-i18n admin
+
+# Start on custom port
+docs-i18n admin --port 4000
+```
+
+The dashboard opens at `http://localhost:3456` (or your custom port). See the [Admin Dashboard](./admin.md) documentation for details on its features.
+
+---
+
+### `site`
+
+Manage the documentation site. Built with TanStack Start and supports dev server, production build, content upload to D1, and deployment to Cloudflare Workers.
+
+```bash
+docs-i18n site [subcommand] [options]
+```
+
+**Subcommands:**
+
+| Subcommand | Description |
+| --- | --- |
+| `dev` (default) | Start the Vite dev server with hot reload |
+| `build` | Build the site for production |
+| `upload` | Upload content and translations to Cloudflare D1 |
+| `deploy` | Deploy to Cloudflare Workers |
+
+**Optional:**
+
+| Flag | Default | Description |
+| --- | --- | --- |
+| `--port <n>` | 3000 | Port for the dev server |
+| `--db <name>` | `docs-i18n-db` | D1 database name (for `upload`) |
+
+**Examples:**
+
+```bash
+# Start dev server
+docs-i18n site
+
+# Start on custom port
+docs-i18n site dev --port 4000
+
+# Build for production
+docs-i18n site build
+
+# Upload content to D1
+docs-i18n site upload --db my-docs-db
+
+# Deploy to Cloudflare Workers
+docs-i18n site deploy
+```
+
+**How it works:**
+
+The `site` command does not require a `docs-i18n.config.ts`. It reads the `site.config.ts` file from your project root to configure the documentation site (project names, colors, locales, etc.).
+
+On first run, the template's dependencies are automatically installed. The dev server uses Vite with the consumer's `site.config.ts` aliased into the template for customization.
+
+For non-English languages, the site automatically loads translations from the `.cache/translations.db` SQLite cache and assembles them at runtime.
+
+---
+
+## Common Workflows
+
+### Full translation pipeline
+
+```bash
+# 1. Scan source files
+docs-i18n rescan
+
+# 2. Check what needs translating
+docs-i18n status
+
+# 3. Translate
+docs-i18n translate --lang zh-hans
+
+# 4. Assemble output
+docs-i18n assemble --lang zh-hans
+```
+
+### Translate a single file for review
+
+```bash
+docs-i18n translate --lang zh-hans --files docs/getting-started.mdx --max 1
+docs-i18n assemble --lang zh-hans
+```
+
+### Preview translation volume before spending API credits
+
+```bash
+docs-i18n translate --lang ja --dry-run
+```
+
+This reports the number of untranslated keys and chunks without making any API calls.
+
+### Multi-project translation
+
+```bash
+# Translate only the "query" project
+docs-i18n translate --lang zh-hans --project query
+
+# Assemble only the "table" project
+docs-i18n assemble --project table --lang zh-hans
+```
+
+### After editing source files
+
+```bash
+# Rescan to detect changes and clean orphans
+docs-i18n rescan
+
+# Translate only new/changed content (cached translations are reused)
+docs-i18n translate --lang zh-hans
+
+# Rebuild output files
+docs-i18n assemble
+```

package/template/content/docs-i18n/en/configuration.md

@@ -0,0 +1,331 @@
+---
+title: Configuration
+description: Complete reference for the docs-i18n configuration file and all available options.
+---
+
+# Configuration
+
+docs-i18n is configured via a `docs-i18n.config.ts` file in your project root. Use the `defineConfig` helper for type safety.
+
+```ts
+import { defineConfig } from 'docs-i18n';
+
+export default defineConfig({
+  // ... options
+});
+```
+
+## Full Config Reference
+
+```ts
+interface DocsI18nConfig {
+  projects: Record<string, ProjectConfig>;
+  languages: string[];
+  cacheDir?: string;
+  translatableFields?: string[];
+  include?: string[];
+  context?: string;
+  llm?: {
+    provider?: 'openrouter' | 'openai' | 'anthropic';
+    model?: string;
+    apiKey?: string;
+    contextLength?: number;
+    maxTokens?: number;
+  };
+  onTranslated?: (info: {
+    project: string;
+    version: string;
+    lang: string;
+  }) => Promise<void>;
+}
+
+interface ProjectConfig {
+  sources: Record<string, string>;
+}
+```
+
+## Options
+
+### `projects`
+
+**Required.** A map of project IDs to their configuration. Each project has a `sources` object mapping version names to source directories (relative to project root).
+
+```ts
+projects: {
+  mydocs: {
+    sources: {
+      latest: 'content/docs',
+    },
+  },
+}
+```
+
+The version key (e.g., `latest`, `v1`) is used to organize translations in the cache and in assembled output. The source path points to the directory containing your English markdown/MDX files.
+
+**Single project:**
+
+When you have a single project, version keys are used directly in the database and output paths (e.g., `latest`).
+
+```ts
+projects: {
+  docs: {
+    sources: { latest: 'content/docs' },
+  },
+}
+```
+
+**Multi-project:**
+
+When you have multiple projects, keys become compound: `project/version` (e.g., `query/latest`, `table/v1`).
+
+```ts
+projects: {
+  query: {
+    sources: {
+      latest: 'content/query/latest',
+      v4: 'content/query/v4',
+    },
+  },
+  table: {
+    sources: {
+      latest: 'content/table/latest',
+    },
+  },
+  blog: {
+    sources: {
+      latest: 'content/blog',
+    },
+  },
+}
+```
+
+### `languages`
+
+**Required.** An array of target language codes to translate into. These codes are used as identifiers in the cache and output directories.
+
+```ts
+languages: ['zh-hans', 'ja', 'es', 'ko', 'fr', 'de']
+```
+
+Common language codes:
+
+| Code | Language |
+| --- | --- |
+| `zh-hans` | Simplified Chinese |
+| `zh-hant` | Traditional Chinese |
+| `ja` | Japanese |
+| `ko` | Korean |
+| `es` | Spanish |
+| `fr` | French |
+| `de` | German |
+| `pt` | Portuguese |
+| `ru` | Russian |
+| `ar` | Arabic |
+| `vi` | Vietnamese |
+| `hi` | Hindi |
+| `th` | Thai |
+
+### `cacheDir`
+
+**Optional.** Directory for the SQLite cache database and assembled output. Default: `'.cache'`.
+
+```ts
+cacheDir: '.cache'
+```
+
+The cache database is stored at `<cacheDir>/translations.db`. Assembled output files are written to `<cacheDir>/content/<version>/<lang>/`.
+
+### `translatableFields`
+
+**Optional.** Frontmatter YAML fields to translate. Supports dot notation for nested fields. Default: `['title', 'description', 'nav_title']`.
+
+```ts
+translatableFields: ['title', 'description', 'nav_title', 'related.title', 'related.description']
+```
+
+When a frontmatter block is detected, only these fields are extracted and sent to the LLM as plain text. All other frontmatter fields are preserved unchanged. After translation, the frontmatter is reconstructed with translated values while maintaining original YAML formatting.
+
+### `include`
+
+**Optional.** Glob patterns for files to include. Default: `['**/*.mdx', '**/*.md']`.
+
+```ts
+include: ['**/*.mdx', '**/*.md']
+```
+
+Files not matching these patterns are ignored during scanning, translation, and assembly.
+
+### `context`
+
+**Optional.** Project context injected into the LLM system prompt. Use this to provide domain-specific information that helps the LLM produce better translations.
+
+```ts
+context: 'TanStack Query is a data-fetching and state management library for React, Vue, Solid, and Angular. It was previously known as React Query.'
+```
+
+This text is appended to the system prompt under a "CONTEXT" section, giving the LLM background about your project, terminology preferences, or naming conventions.
+
+### `llm`
+
+**Optional.** LLM provider configuration.
+
+```ts
+llm: {
+  provider: 'openrouter',
+  model: 'qwen/qwen3.5-flash-02-23',
+  apiKey: process.env.OPENROUTER_API_KEY,
+  contextLength: 1_000_000,
+  maxTokens: 65536,
+}
+```
+
+#### `llm.provider`
+
+The API provider. Default: `'openrouter'`.
+
+| Provider | Base URL | API Key Env Var |
+| --- | --- | --- |
+| `'openrouter'` | `https://openrouter.ai/api/v1` | `OPENROUTER_API_KEY` |
+| `'openai'` | `https://api.deepseek.com` | `OPENAI_API_KEY` |
+| `'anthropic'` | `https://api.anthropic.com/v1` | `ANTHROPIC_API_KEY` |
+
+All providers use the OpenAI-compatible chat completions API.
+
+#### `llm.model`
+
+The model identifier. Defaults vary by provider:
+
+- `openrouter`: `'deepseek/deepseek-chat-v3-0324:free'`
+- `openai`: `'deepseek-chat'`
+- `anthropic`: `'claude-sonnet-4-20250514'`
+
+You can also override the model per command with `--model`.
+
+#### `llm.apiKey`
+
+API key for the provider. Falls back to the corresponding environment variable (`OPENROUTER_API_KEY`, `OPENAI_API_KEY`, or `ANTHROPIC_API_KEY`). Can also be overridden per command with `--api-key`.
+
+#### `llm.contextLength`
+
+Total context window of the model in tokens. Default: `32768`. Used to calculate how many translation nodes fit in a single API call. Set this to match your model's actual context window for optimal chunking.
+
+#### `llm.maxTokens`
+
+Maximum output tokens per API call. Default: `16384`. The actual output budget is calculated as `min(maxTokens, contextLength - 4096)`. The translator detects truncated output (when `finish_reason` is `'length'`) and retries.
+
+### `onTranslated`
+
+**Optional.** Async callback invoked after translation completes for a project/version/language combination.
+
+```ts
+onTranslated: async ({ project, version, lang }) => {
+  console.log(`Finished translating ${project}/${version} to ${lang}`);
+  // Trigger a build, send a notification, etc.
+}
+```
+
+## Example Configurations
+
+### Simple single-project setup
+
+```ts
+import { defineConfig } from 'docs-i18n';
+
+export default defineConfig({
+  projects: {
+    docs: {
+      sources: { latest: 'docs' },
+    },
+  },
+  languages: ['zh-hans'],
+  llm: {
+    provider: 'openrouter',
+    apiKey: process.env.OPENROUTER_API_KEY,
+    model: 'deepseek/deepseek-chat-v3-0324:free',
+  },
+});
+```
+
+### Multi-project monorepo (e.g., TanStack)
+
+```ts
+import { defineConfig } from 'docs-i18n';
+
+export default defineConfig({
+  projects: {
+    query: {
+      sources: {
+        latest: 'content/query/latest',
+        v4: 'content/query/v4',
+      },
+    },
+    table: {
+      sources: { latest: 'content/table/latest' },
+    },
+    router: {
+      sources: { latest: 'content/router/latest' },
+    },
+    blog: {
+      sources: { latest: 'content/blog' },
+    },
+  },
+  languages: ['zh-hans', 'ja', 'ko', 'es'],
+  cacheDir: '.cache',
+  translatableFields: ['title', 'description', 'nav_title'],
+  include: ['**/*.md'],
+  context: 'TanStack provides high-quality open-source libraries for web development including Query, Router, Table, and Form.',
+  llm: {
+    provider: 'openrouter',
+    model: 'qwen/qwen3.5-flash-02-23',
+    apiKey: process.env.OPENROUTER_API_KEY,
+    contextLength: 1_000_000,
+    maxTokens: 65536,
+  },
+});
+```
+
+### Using Anthropic
+
+```ts
+import { defineConfig } from 'docs-i18n';
+
+export default defineConfig({
+  projects: {
+    docs: {
+      sources: { latest: 'content/docs' },
+    },
+  },
+  languages: ['ja', 'fr'],
+  llm: {
+    provider: 'anthropic',
+    model: 'claude-sonnet-4-20250514',
+    apiKey: process.env.ANTHROPIC_API_KEY,
+    contextLength: 200000,
+    maxTokens: 16384,
+  },
+});
+```
+
+### With post-translation hook
+
+```ts
+import { defineConfig } from 'docs-i18n';
+import { exec } from 'node:child_process';
+
+export default defineConfig({
+  projects: {
+    docs: {
+      sources: { latest: 'content/docs' },
+    },
+  },
+  languages: ['zh-hans'],
+  llm: {
+    provider: 'openrouter',
+    apiKey: process.env.OPENROUTER_API_KEY,
+  },
+  onTranslated: async ({ project, version, lang }) => {
+    // Rebuild the site after translation
+    exec('npm run build');
+  },
+});
+```