docs-i18n 0.8.1 → 0.8.3

package/template/content/docs-i18n/en/configuration.md

@@ -0,0 +1,350 @@
+---
+title: Configuration
+description: Complete reference for the docs-i18n configuration file and all available options.
+---
+
+# Configuration
+
+docs-i18n is configured via a `docs-i18n.config.ts` file in your project root. Use the `defineConfig` helper for type safety.
+
+```ts
+import { defineConfig } from 'docs-i18n';
+
+export default defineConfig({
+  // ... options
+});
+```
+
+## Full Config Reference
+
+```ts
+interface DocsI18nConfig {
+  projects: Record<string, ProjectConfig>;
+  languages: string[];
+  cacheDir?: string;
+  translatableFields?: string[];
+  include?: string[];
+  context?: string;
+  llm?: {
+    provider?: 'openrouter' | 'openai' | 'anthropic';
+    model?: string;
+    apiKey?: string;
+    contextLength?: number;
+    maxTokens?: number;
+  };
+  onTranslated?: (info: {
+    project: string;
+    version: string;
+    lang: string;
+  }) => Promise<void>;
+}
+
+interface ProjectConfig {
+  sources: Record<string, string>;
+}
+```
+
+## Options
+
+### `projects`
+
+**Required.** A map of project IDs to their configuration. Each project has a `sources` object mapping version names to source directories (relative to project root).
+
+```ts
+projects: {
+  mydocs: {
+    sources: {
+      latest: 'content/docs',
+    },
+  },
+}
+```
+
+The version key (e.g., `latest`, `v1`) is used to organize translations in the cache and in assembled output. The source path points to the directory containing your English markdown/MDX files.
+
+```
+Content directory convention:
+
+  content/                        ← English source (no /en/ subdir)
+    {project}/{version}/          ← source path in config
+      file.md
+      subdir/file.mdx
+
+  .cache/
+    translations.db               ← all translations
+    content/{version}/{lang}/     ← assembled output
+```
+
+**Single project:**
+
+```ts
+projects: {
+  docs: {
+    sources: { latest: 'content/docs' },
+  },
+}
+```
+
+```
+content/docs/              ← source files (English)
+  getting-started.md
+  api-reference.md
+```
+
+**Multi-project:**
+
+When you have multiple projects, keys become compound: `project/version` (e.g., `query/latest`, `table/v1`).
+
+```ts
+projects: {
+  query: {
+    sources: {
+      v5: 'content/query/v5',
+      v4: 'content/query/v4',
+    },
+  },
+  table: {
+    sources: {
+      latest: 'content/table/latest',
+    },
+  },
+}
+```
+
+```
+content/
+  query/v5/overview.md     ← query project, v5
+  query/v4/overview.md     ← query project, v4
+  table/latest/overview.md ← table project
+```
+
+### `languages`
+
+**Required.** An array of target language codes to translate into. These codes are used as identifiers in the cache and output directories.
+
+```ts
+languages: ['zh-hans', 'ja', 'es', 'ko', 'fr', 'de']
+```
+
+Common language codes:
+
+| Code | Language |
+| --- | --- |
+| `zh-hans` | Simplified Chinese |
+| `zh-hant` | Traditional Chinese |
+| `ja` | Japanese |
+| `ko` | Korean |
+| `es` | Spanish |
+| `fr` | French |
+| `de` | German |
+| `pt` | Portuguese |
+| `ru` | Russian |
+| `ar` | Arabic |
+| `vi` | Vietnamese |
+| `hi` | Hindi |
+| `th` | Thai |
+
+### `cacheDir`
+
+**Optional.** Directory for the SQLite cache database and assembled output. Default: `'.cache'`.
+
+```ts
+cacheDir: '.cache'
+```
+
+The cache database is stored at `<cacheDir>/translations.db`. Assembled output files are written to `<cacheDir>/content/<version>/<lang>/`.
+
+### `translatableFields`
+
+**Optional.** Frontmatter YAML fields to translate. Supports dot notation for nested fields. Default: `['title', 'description', 'nav_title']`.
+
+```ts
+translatableFields: ['title', 'description', 'nav_title', 'related.title', 'related.description']
+```
+
+When a frontmatter block is detected, only these fields are extracted and sent to the LLM as plain text. All other frontmatter fields are preserved unchanged. After translation, the frontmatter is reconstructed with translated values while maintaining original YAML formatting.
+
+### `include`
+
+**Optional.** Glob patterns for files to include. Default: `['**/*.mdx', '**/*.md']`.
+
+```ts
+include: ['**/*.mdx', '**/*.md']
+```
+
+Files not matching these patterns are ignored during scanning, translation, and assembly.
+
+### `context`
+
+**Optional.** Project context injected into the LLM system prompt. Use this to provide domain-specific information that helps the LLM produce better translations.
+
+```ts
+context: 'TanStack Query is a data-fetching and state management library for React, Vue, Solid, and Angular. It was previously known as React Query.'
+```
+
+This text is appended to the system prompt under a "CONTEXT" section, giving the LLM background about your project, terminology preferences, or naming conventions.
+
+### `llm`
+
+**Optional.** LLM provider configuration.
+
+```ts
+llm: {
+  provider: 'openrouter',
+  model: 'qwen/qwen3.5-flash-02-23',
+  apiKey: process.env.OPENROUTER_API_KEY,
+  contextLength: 1_000_000,
+  maxTokens: 65536,
+}
+```
+
+#### `llm.provider`
+
+The API provider. Default: `'openrouter'`.
+
+| Provider | Base URL | API Key Env Var |
+| --- | --- | --- |
+| `'openrouter'` | `https://openrouter.ai/api/v1` | `OPENROUTER_API_KEY` |
+| `'openai'` | `https://api.deepseek.com` | `OPENAI_API_KEY` |
+| `'anthropic'` | `https://api.anthropic.com/v1` | `ANTHROPIC_API_KEY` |
+
+All providers use the OpenAI-compatible chat completions API.
+
+#### `llm.model`
+
+The model identifier. Defaults vary by provider:
+
+- `openrouter`: `'deepseek/deepseek-chat-v3-0324:free'`
+- `openai`: `'deepseek-chat'`
+- `anthropic`: `'claude-sonnet-4-20250514'`
+
+You can also override the model per command with `--model`.
+
+#### `llm.apiKey`
+
+API key for the provider. Falls back to the corresponding environment variable (`OPENROUTER_API_KEY`, `OPENAI_API_KEY`, or `ANTHROPIC_API_KEY`). Can also be overridden per command with `--api-key`.
+
+#### `llm.contextLength`
+
+Total context window of the model in tokens. Default: `32768`. Used to calculate how many translation nodes fit in a single API call. Set this to match your model's actual context window for optimal chunking.
+
+#### `llm.maxTokens`
+
+Maximum output tokens per API call. Default: `16384`. The actual output budget is calculated as `min(maxTokens, contextLength - 4096)`. The translator detects truncated output (when `finish_reason` is `'length'`) and retries.
+
+### `onTranslated`
+
+**Optional.** Async callback invoked after translation completes for a project/version/language combination.
+
+```ts
+onTranslated: async ({ project, version, lang }) => {
+  console.log(`Finished translating ${project}/${version} to ${lang}`);
+  // Trigger a build, send a notification, etc.
+}
+```
+
+## Example Configurations
+
+### Simple single-project setup
+
+```ts
+import { defineConfig } from 'docs-i18n';
+
+export default defineConfig({
+  projects: {
+    docs: {
+      sources: { latest: 'docs' },
+    },
+  },
+  languages: ['zh-hans'],
+  llm: {
+    provider: 'openrouter',
+    apiKey: process.env.OPENROUTER_API_KEY,
+    model: 'deepseek/deepseek-chat-v3-0324:free',
+  },
+});
+```
+
+### Multi-project monorepo (e.g., TanStack)
+
+```ts
+import { defineConfig } from 'docs-i18n';
+
+export default defineConfig({
+  projects: {
+    query: {
+      sources: {
+        latest: 'content/query/latest',
+        v4: 'content/query/v4',
+      },
+    },
+    table: {
+      sources: { latest: 'content/table/latest' },
+    },
+    router: {
+      sources: { latest: 'content/router/latest' },
+    },
+    blog: {
+      sources: { latest: 'content/blog' },
+    },
+  },
+  languages: ['zh-hans', 'ja', 'ko', 'es'],
+  cacheDir: '.cache',
+  translatableFields: ['title', 'description', 'nav_title'],
+  include: ['**/*.md'],
+  context: 'TanStack provides high-quality open-source libraries for web development including Query, Router, Table, and Form.',
+  llm: {
+    provider: 'openrouter',
+    model: 'qwen/qwen3.5-flash-02-23',
+    apiKey: process.env.OPENROUTER_API_KEY,
+    contextLength: 1_000_000,
+    maxTokens: 65536,
+  },
+});
+```
+
+### Using Anthropic
+
+```ts
+import { defineConfig } from 'docs-i18n';
+
+export default defineConfig({
+  projects: {
+    docs: {
+      sources: { latest: 'content/docs' },
+    },
+  },
+  languages: ['ja', 'fr'],
+  llm: {
+    provider: 'anthropic',
+    model: 'claude-sonnet-4-20250514',
+    apiKey: process.env.ANTHROPIC_API_KEY,
+    contextLength: 200000,
+    maxTokens: 16384,
+  },
+});
+```
+
+### With post-translation hook
+
+```ts
+import { defineConfig } from 'docs-i18n';
+import { exec } from 'node:child_process';
+
+export default defineConfig({
+  projects: {
+    docs: {
+      sources: { latest: 'content/docs' },
+    },
+  },
+  languages: ['zh-hans'],
+  llm: {
+    provider: 'openrouter',
+    apiKey: process.env.OPENROUTER_API_KEY,
+  },
+  onTranslated: async ({ project, version, lang }) => {
+    // Rebuild the site after translation
+    exec('npm run build');
+  },
+});
+```

package/template/content/docs-i18n/en/deployment.md

@@ -0,0 +1,222 @@
+---
+title: Deployment
+description: Using docs-i18n in CI/CD pipelines, deploying the admin dashboard, and runtime translation with Cloudflare D1.
+---
+
+# Deployment
+
+## CI/CD with GitHub Actions
+
+docs-i18n works well in CI/CD pipelines. Since translations are cached in SQLite, you only pay for API calls on new or changed content.
+
+### Basic translation workflow
+
+```yaml
+name: Translate Docs
+on:
+  push:
+    branches: [main]
+    paths:
+      - 'content/**'
+
+jobs:
+  translate:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 20
+
+      - run: npm ci
+
+      # Restore the SQLite cache from a previous run
+      - uses: actions/cache@v4
+        with:
+          path: .cache
+          key: i18n-cache-${{ hashFiles('content/**') }}
+          restore-keys: |
+            i18n-cache-
+
+      # Rescan source files to detect changes
+      - run: npx docs-i18n rescan
+
+      # Translate all configured languages
+      - run: npx docs-i18n translate --lang zh-hans
+        env:
+          OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
+
+      - run: npx docs-i18n translate --lang ja
+        env:
+          OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
+
+      # Assemble output files
+      - run: npx docs-i18n assemble
+
+      # Commit translated files (or use them in a build step)
+      - uses: stefanzweifel/git-auto-commit-action@v5
+        with:
+          commit_message: 'chore: update translations'
+          file_pattern: '.cache/content/**'
+```
+
+### Tips for CI/CD
+
+- **Cache the SQLite database.** The `.cache/translations.db` file contains all cached translations. Restoring it between runs avoids re-translating unchanged content. Use `actions/cache` with a key based on your content files.
+- **Run `rescan` before `translate`.** This detects new, changed, and deleted source files, and cleans up orphaned translations.
+- **Use `--dry-run` in PR checks.** Add a CI step that runs `docs-i18n translate --lang zh-hans --dry-run` to report how many nodes need translation without spending API credits.
+- **Limit concurrency.** In CI, you may want `--concurrency 1` to avoid rate limiting from your LLM provider.
+- **Set `--max` for budgeting.** Use `--max 10` to cap the number of API calls per run if you want to translate incrementally across multiple CI runs.
+
+### PR status check
+
+```yaml
+name: Translation Status
+on:
+  pull_request:
+    paths:
+      - 'content/**'
+
+jobs:
+  check:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 20
+      - run: npm ci
+
+      - uses: actions/cache@v4
+        with:
+          path: .cache
+          key: i18n-cache-${{ hashFiles('content/**') }}
+          restore-keys: |
+            i18n-cache-
+
+      - run: npx docs-i18n rescan
+      - run: npx docs-i18n status
+      - run: npx docs-i18n translate --lang zh-hans --dry-run
+```
+
+## Deployment Architecture
+
+```
+┌──────────────────────────────────────────────────────────┐
+│  Development (local)                                      │
+│                                                          │
+│  docs-i18n admin ──→ Pre-built Node.js server (port 3456)│
+│  docs-i18n site  ──→ Vite dev server (port 3000)         │
+│                      ↕ reads content/ + .cache/           │
+└──────────────────────────────────────────────────────────┘
+
+┌──────────────────────────────────────────────────────────┐
+│  Production (Cloudflare Workers)                          │
+│                                                          │
+│  docs-i18n site build ──→ .output/                       │
+│  docs-i18n site upload ──→ D1 (translations)             │
+│  docs-i18n site deploy ──→ CF Workers (SSR)              │
+│                                                          │
+│  Request → Worker → fetch EN content → D1 translate      │
+│                   → SSR render → response                │
+└──────────────────────────────────────────────────────────┘
+```
+
+## Admin Dashboard
+
+The admin dashboard is pre-built and requires no additional dependencies. It runs as a local Node.js server.
+
+```bash
+npx docs-i18n admin
+# or
+npx docs-i18n admin --port 4000
+```
+
+It reads your `docs-i18n.config.ts` to discover projects, versions, and languages.
+
+## Runtime Translation with D1
+
+For SSR sites that fetch markdown at runtime (e.g., TanStack Start on Cloudflare Workers), docs-i18n provides a lightweight runtime translator that queries Cloudflare D1 for cached translations.
+
+### How it works
+
+The `createTranslator` function from `docs-i18n/serve` creates a translator instance. When `translate()` is called:
+
+1. The markdown is parsed into AST nodes using the same parser as the CLI.
+2. Translatable nodes' MD5 keys are collected.
+3. A batch query is sent to D1 to look up all translations for the given language.
+4. The content is reassembled: translated nodes use the D1 cache value, untranslated nodes fall back to the original English text.
+
+### Setup
+
+#### 1. Export translations to D1
+
+First, translate your content using the CLI as normal. Then export the SQLite cache to D1. You can use Wrangler to create a D1 database and import the translations:
+
+```bash
+# Create a D1 database
+wrangler d1 create docs-translations
+
+# Export the translations table from your local SQLite
+sqlite3 .cache/translations.db ".dump translations" > translations.sql
+
+# Import into D1
+wrangler d1 execute docs-translations --file=translations.sql
+```
+
+Make sure the D1 database has the same `translations` table schema:
+
+```sql
+CREATE TABLE IF NOT EXISTS translations (
+  lang       TEXT NOT NULL,
+  key        TEXT NOT NULL,
+  value      TEXT NOT NULL,
+  created_at INTEGER NOT NULL DEFAULT (unixepoch()),
+  updated_at INTEGER NOT NULL DEFAULT (unixepoch()),
+  PRIMARY KEY (lang, key)
+);
+```
+
+#### 2. Use the translator in your Worker
+
+```ts
+import { createTranslator } from 'docs-i18n/serve';
+
+const translator = createTranslator();
+
+// In your request handler (e.g., TanStack Start server function):
+export async function getTranslatedDoc(
+  repo: string,
+  branch: string,
+  filePath: string,
+  lang: string,
+  env: { DB: D1Database },
+) {
+  // Fetch the English markdown from GitHub
+  const enMarkdown = await fetchFromGitHub(repo, branch, filePath);
+
+  // If the requested language is English, return as-is
+  if (lang === 'en') return enMarkdown;
+
+  // Translate using D1 cache
+  const translated = await translator.translate(enMarkdown, lang, env.DB);
+  return translated;
+}
+```
+
+#### 3. Bind D1 in wrangler.toml
+
+```toml
+[[d1_databases]]
+binding = "DB"
+database_name = "docs-translations"
+database_id = "your-database-id"
+```
+
+### Performance considerations
+
+- The translator uses a single batch query to D1, so latency scales with the number of translatable nodes (typically under 50ms for a standard documentation page).
+- English content (`lang === 'en'`) short-circuits immediately without parsing or querying.
+- The parser runs on every request since the English source is fetched at runtime. For heavy traffic, consider caching the translated output at the edge.
+- Untranslated nodes gracefully fall back to English, so partial translations work without errors.