docs-i18n 0.8.1 → 0.8.2

package/template/content/en/configuration.md

@@ -0,0 +1,331 @@
+---
+title: Configuration
+description: Complete reference for the docs-i18n configuration file and all available options.
+---
+
+# Configuration
+
+docs-i18n is configured via a `docs-i18n.config.ts` file in your project root. Use the `defineConfig` helper for type safety.
+
+```ts
+import { defineConfig } from 'docs-i18n';
+
+export default defineConfig({
+  // ... options
+});
+```
+
+## Full Config Reference
+
+```ts
+interface DocsI18nConfig {
+  projects: Record<string, ProjectConfig>;
+  languages: string[];
+  cacheDir?: string;
+  translatableFields?: string[];
+  include?: string[];
+  context?: string;
+  llm?: {
+    provider?: 'openrouter' | 'openai' | 'anthropic';
+    model?: string;
+    apiKey?: string;
+    contextLength?: number;
+    maxTokens?: number;
+  };
+  onTranslated?: (info: {
+    project: string;
+    version: string;
+    lang: string;
+  }) => Promise<void>;
+}
+
+interface ProjectConfig {
+  sources: Record<string, string>;
+}
+```
+
+## Options
+
+### `projects`
+
+**Required.** A map of project IDs to their configuration. Each project has a `sources` object mapping version names to source directories (relative to project root).
+
+```ts
+projects: {
+  mydocs: {
+    sources: {
+      latest: 'content/docs',
+    },
+  },
+}
+```
+
+The version key (e.g., `latest`, `v1`) is used to organize translations in the cache and in assembled output. The source path points to the directory containing your English markdown/MDX files.
+
+**Single project:**
+
+When you have a single project, version keys are used directly in the database and output paths (e.g., `latest`).
+
+```ts
+projects: {
+  docs: {
+    sources: { latest: 'content/docs' },
+  },
+}
+```
+
+**Multi-project:**
+
+When you have multiple projects, keys become compound: `project/version` (e.g., `query/latest`, `table/v1`).
+
+```ts
+projects: {
+  query: {
+    sources: {
+      latest: 'content/query/latest',
+      v4: 'content/query/v4',
+    },
+  },
+  table: {
+    sources: {
+      latest: 'content/table/latest',
+    },
+  },
+  blog: {
+    sources: {
+      latest: 'content/blog',
+    },
+  },
+}
+```
+
+### `languages`
+
+**Required.** An array of target language codes to translate into. These codes are used as identifiers in the cache and output directories.
+
+```ts
+languages: ['zh-hans', 'ja', 'es', 'ko', 'fr', 'de']
+```
+
+Common language codes:
+
+| Code | Language |
+| --- | --- |
+| `zh-hans` | Simplified Chinese |
+| `zh-hant` | Traditional Chinese |
+| `ja` | Japanese |
+| `ko` | Korean |
+| `es` | Spanish |
+| `fr` | French |
+| `de` | German |
+| `pt` | Portuguese |
+| `ru` | Russian |
+| `ar` | Arabic |
+| `vi` | Vietnamese |
+| `hi` | Hindi |
+| `th` | Thai |
+
+### `cacheDir`
+
+**Optional.** Directory for the SQLite cache database and assembled output. Default: `'.cache'`.
+
+```ts
+cacheDir: '.cache'
+```
+
+The cache database is stored at `<cacheDir>/translations.db`. Assembled output files are written to `<cacheDir>/content/<version>/<lang>/`.
+
+### `translatableFields`
+
+**Optional.** Frontmatter YAML fields to translate. Supports dot notation for nested fields. Default: `['title', 'description', 'nav_title']`.
+
+```ts
+translatableFields: ['title', 'description', 'nav_title', 'related.title', 'related.description']
+```
+
+When a frontmatter block is detected, only these fields are extracted and sent to the LLM as plain text. All other frontmatter fields are preserved unchanged. After translation, the frontmatter is reconstructed with translated values while maintaining original YAML formatting.
+
+### `include`
+
+**Optional.** Glob patterns for files to include. Default: `['**/*.mdx', '**/*.md']`.
+
+```ts
+include: ['**/*.mdx', '**/*.md']
+```
+
+Files not matching these patterns are ignored during scanning, translation, and assembly.
+
+### `context`
+
+**Optional.** Project context injected into the LLM system prompt. Use this to provide domain-specific information that helps the LLM produce better translations.
+
+```ts
+context: 'TanStack Query is a data-fetching and state management library for React, Vue, Solid, and Angular. It was previously known as React Query.'
+```
+
+This text is appended to the system prompt under a "CONTEXT" section, giving the LLM background about your project, terminology preferences, or naming conventions.
+
+### `llm`
+
+**Optional.** LLM provider configuration.
+
+```ts
+llm: {
+  provider: 'openrouter',
+  model: 'qwen/qwen3.5-flash-02-23',
+  apiKey: process.env.OPENROUTER_API_KEY,
+  contextLength: 1_000_000,
+  maxTokens: 65536,
+}
+```
+
+#### `llm.provider`
+
+The API provider. Default: `'openrouter'`.
+
+| Provider | Base URL | API Key Env Var |
+| --- | --- | --- |
+| `'openrouter'` | `https://openrouter.ai/api/v1` | `OPENROUTER_API_KEY` |
+| `'openai'` | `https://api.deepseek.com` | `OPENAI_API_KEY` |
+| `'anthropic'` | `https://api.anthropic.com/v1` | `ANTHROPIC_API_KEY` |
+
+All providers use the OpenAI-compatible chat completions API.
+
+#### `llm.model`
+
+The model identifier. Defaults vary by provider:
+
+- `openrouter`: `'deepseek/deepseek-chat-v3-0324:free'`
+- `openai`: `'deepseek-chat'`
+- `anthropic`: `'claude-sonnet-4-20250514'`
+
+You can also override the model per command with `--model`.
+
+#### `llm.apiKey`
+
+API key for the provider. Falls back to the corresponding environment variable (`OPENROUTER_API_KEY`, `OPENAI_API_KEY`, or `ANTHROPIC_API_KEY`). Can also be overridden per command with `--api-key`.
+
+#### `llm.contextLength`
+
+Total context window of the model in tokens. Default: `32768`. Used to calculate how many translation nodes fit in a single API call. Set this to match your model's actual context window for optimal chunking.
+
+#### `llm.maxTokens`
+
+Maximum output tokens per API call. Default: `16384`. The actual output budget is calculated as `min(maxTokens, contextLength - 4096)`. The translator detects truncated output (when `finish_reason` is `'length'`) and retries.
+
+### `onTranslated`
+
+**Optional.** Async callback invoked after translation completes for a project/version/language combination.
+
+```ts
+onTranslated: async ({ project, version, lang }) => {
+  console.log(`Finished translating ${project}/${version} to ${lang}`);
+  // Trigger a build, send a notification, etc.
+}
+```
+
+## Example Configurations
+
+### Simple single-project setup
+
+```ts
+import { defineConfig } from 'docs-i18n';
+
+export default defineConfig({
+  projects: {
+    docs: {
+      sources: { latest: 'docs' },
+    },
+  },
+  languages: ['zh-hans'],
+  llm: {
+    provider: 'openrouter',
+    apiKey: process.env.OPENROUTER_API_KEY,
+    model: 'deepseek/deepseek-chat-v3-0324:free',
+  },
+});
+```
+
+### Multi-project monorepo (e.g., TanStack)
+
+```ts
+import { defineConfig } from 'docs-i18n';
+
+export default defineConfig({
+  projects: {
+    query: {
+      sources: {
+        latest: 'content/query/latest',
+        v4: 'content/query/v4',
+      },
+    },
+    table: {
+      sources: { latest: 'content/table/latest' },
+    },
+    router: {
+      sources: { latest: 'content/router/latest' },
+    },
+    blog: {
+      sources: { latest: 'content/blog' },
+    },
+  },
+  languages: ['zh-hans', 'ja', 'ko', 'es'],
+  cacheDir: '.cache',
+  translatableFields: ['title', 'description', 'nav_title'],
+  include: ['**/*.md'],
+  context: 'TanStack provides high-quality open-source libraries for web development including Query, Router, Table, and Form.',
+  llm: {
+    provider: 'openrouter',
+    model: 'qwen/qwen3.5-flash-02-23',
+    apiKey: process.env.OPENROUTER_API_KEY,
+    contextLength: 1_000_000,
+    maxTokens: 65536,
+  },
+});
+```
+
+### Using Anthropic
+
+```ts
+import { defineConfig } from 'docs-i18n';
+
+export default defineConfig({
+  projects: {
+    docs: {
+      sources: { latest: 'content/docs' },
+    },
+  },
+  languages: ['ja', 'fr'],
+  llm: {
+    provider: 'anthropic',
+    model: 'claude-sonnet-4-20250514',
+    apiKey: process.env.ANTHROPIC_API_KEY,
+    contextLength: 200000,
+    maxTokens: 16384,
+  },
+});
+```
+
+### With post-translation hook
+
+```ts
+import { defineConfig } from 'docs-i18n';
+import { exec } from 'node:child_process';
+
+export default defineConfig({
+  projects: {
+    docs: {
+      sources: { latest: 'content/docs' },
+    },
+  },
+  languages: ['zh-hans'],
+  llm: {
+    provider: 'openrouter',
+    apiKey: process.env.OPENROUTER_API_KEY,
+  },
+  onTranslated: async ({ project, version, lang }) => {
+    // Rebuild the site after translation
+    exec('npm run build');
+  },
+});
+```

package/template/content/en/deployment.md

@@ -0,0 +1,209 @@
+---
+title: Deployment
+description: Using docs-i18n in CI/CD pipelines, deploying the admin dashboard, and runtime translation with Cloudflare D1.
+---
+
+# Deployment
+
+## CI/CD with GitHub Actions
+
+docs-i18n works well in CI/CD pipelines. Since translations are cached in SQLite, you only pay for API calls on new or changed content.
+
+### Basic translation workflow
+
+```yaml
+name: Translate Docs
+on:
+  push:
+    branches: [main]
+    paths:
+      - 'content/**'
+
+jobs:
+  translate:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 20
+
+      - run: npm ci
+
+      # Restore the SQLite cache from a previous run
+      - uses: actions/cache@v4
+        with:
+          path: .cache
+          key: i18n-cache-${{ hashFiles('content/**') }}
+          restore-keys: |
+            i18n-cache-
+
+      # Rescan source files to detect changes
+      - run: npx docs-i18n rescan
+
+      # Translate all configured languages
+      - run: npx docs-i18n translate --lang zh-hans
+        env:
+          OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
+
+      - run: npx docs-i18n translate --lang ja
+        env:
+          OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
+
+      # Assemble output files
+      - run: npx docs-i18n assemble
+
+      # Commit translated files (or use them in a build step)
+      - uses: stefanzweifel/git-auto-commit-action@v5
+        with:
+          commit_message: 'chore: update translations'
+          file_pattern: '.cache/content/**'
+```
+
+### Tips for CI/CD
+
+- **Cache the SQLite database.** The `.cache/translations.db` file contains all cached translations. Restoring it between runs avoids re-translating unchanged content. Use `actions/cache` with a key based on your content files.
+- **Run `rescan` before `translate`.** This detects new, changed, and deleted source files, and cleans up orphaned translations.
+- **Use `--dry-run` in PR checks.** Add a CI step that runs `docs-i18n translate --lang zh-hans --dry-run` to report how many nodes need translation without spending API credits.
+- **Limit concurrency.** In CI, you may want `--concurrency 1` to avoid rate limiting from your LLM provider.
+- **Set `--max` for budgeting.** Use `--max 10` to cap the number of API calls per run if you want to translate incrementally across multiple CI runs.
+
+### PR status check
+
+```yaml
+name: Translation Status
+on:
+  pull_request:
+    paths:
+      - 'content/**'
+
+jobs:
+  check:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 20
+      - run: npm ci
+
+      - uses: actions/cache@v4
+        with:
+          path: .cache
+          key: i18n-cache-${{ hashFiles('content/**') }}
+          restore-keys: |
+            i18n-cache-
+
+      - run: npx docs-i18n rescan
+      - run: npx docs-i18n status
+      - run: npx docs-i18n translate --lang zh-hans --dry-run
+```
+
+## Admin Dashboard Deployment
+
+The admin dashboard is a TanStack Start application that runs locally. It is designed for development use, not production deployment.
+
+To start it:
+
+```bash
+npx docs-i18n admin
+# or
+npx docs-i18n admin --port 4000
+```
+
+The dashboard starts a Vite dev server pointed at the `src/admin` directory within the docs-i18n package. It reads your `docs-i18n.config.ts` to discover projects, versions, and languages.
+
+**Prerequisites:**
+
+Your project must have `vite` and `@vitejs/plugin-react` installed as dev dependencies:
+
+```bash
+npm install -D vite @vitejs/plugin-react
+```
+
+## Runtime Translation with D1
+
+For SSR sites that fetch markdown at runtime (e.g., TanStack Start on Cloudflare Workers), docs-i18n provides a lightweight runtime translator that queries Cloudflare D1 for cached translations.
+
+### How it works
+
+The `createTranslator` function from `docs-i18n/serve` creates a translator instance. When `translate()` is called:
+
+1. The markdown is parsed into AST nodes using the same parser as the CLI.
+2. Translatable nodes' MD5 keys are collected.
+3. A batch query is sent to D1 to look up all translations for the given language.
+4. The content is reassembled: translated nodes use the D1 cache value, untranslated nodes fall back to the original English text.
+
+### Setup
+
+#### 1. Export translations to D1
+
+First, translate your content using the CLI as normal. Then export the SQLite cache to D1. You can use Wrangler to create a D1 database and import the translations:
+
+```bash
+# Create a D1 database
+wrangler d1 create docs-translations
+
+# Export the translations table from your local SQLite
+sqlite3 .cache/translations.db ".dump translations" > translations.sql
+
+# Import into D1
+wrangler d1 execute docs-translations --file=translations.sql
+```
+
+Make sure the D1 database has the same `translations` table schema:
+
+```sql
+CREATE TABLE IF NOT EXISTS translations (
+  lang       TEXT NOT NULL,
+  key        TEXT NOT NULL,
+  value      TEXT NOT NULL,
+  created_at INTEGER NOT NULL DEFAULT (unixepoch()),
+  updated_at INTEGER NOT NULL DEFAULT (unixepoch()),
+  PRIMARY KEY (lang, key)
+);
+```
+
+#### 2. Use the translator in your Worker
+
+```ts
+import { createTranslator } from 'docs-i18n/serve';
+
+const translator = createTranslator();
+
+// In your request handler (e.g., TanStack Start server function):
+export async function getTranslatedDoc(
+  repo: string,
+  branch: string,
+  filePath: string,
+  lang: string,
+  env: { DB: D1Database },
+) {
+  // Fetch the English markdown from GitHub
+  const enMarkdown = await fetchFromGitHub(repo, branch, filePath);
+
+  // If the requested language is English, return as-is
+  if (lang === 'en') return enMarkdown;
+
+  // Translate using D1 cache
+  const translated = await translator.translate(enMarkdown, lang, env.DB);
+  return translated;
+}
+```
+
+#### 3. Bind D1 in wrangler.toml
+
+```toml
+[[d1_databases]]
+binding = "DB"
+database_name = "docs-translations"
+database_id = "your-database-id"
+```
+
+### Performance considerations
+
+- The translator uses a single batch query to D1, so latency scales with the number of translatable nodes (typically under 50ms for a standard documentation page).
+- English content (`lang === 'en'`) short-circuits immediately without parsing or querying.
+- The parser runs on every request since the English source is fetched at runtime. For heavy traffic, consider caching the translated output at the edge.
+- Untranslated nodes gracefully fall back to English, so partial translations work without errors.

package/template/content/en/getting-started.md

@@ -0,0 +1,168 @@
+---
+title: Getting Started
+description: Install docs-i18n and translate your first documentation file in minutes.
+---
+
+# Getting Started
+
+## What is docs-i18n?
+
+docs-i18n is a universal documentation translation engine. It parses markdown and MDX files into translatable AST nodes, translates them via LLM, caches results in SQLite, and assembles translated output files. It is framework-agnostic and works with any documentation site built on Next.js, Astro, TanStack Start, or similar tools.
+
+Key characteristics:
+
+- **Incremental** -- only changed or new content is sent to the LLM.
+- **AST-based** -- uses remark to parse markdown into nodes, producing stable MD5 keys for deduplication.
+- **SQLite-backed** -- concurrent-safe cache with WAL mode. No manual save steps.
+- **Multi-project** -- supports multiple documentation projects and versions in a single config.
+- **Admin dashboard** -- web UI for monitoring progress, managing jobs, and previewing translations.
+- **Runtime serve** -- optional D1-compatible translator for SSR sites on Cloudflare Workers.
+
+## Requirements
+
+- Node.js 20+ or Bun
+- An API key for an LLM provider (OpenRouter, OpenAI, or Anthropic)
+- For the admin dashboard: `vite` and `@vitejs/plugin-react` as dev dependencies
+
+## Installation
+
+```bash
+npm install docs-i18n
+```
+
+Or with Bun:
+
+```bash
+bun add docs-i18n
+```
+
+## Quick Start
+
+### 1. Create a configuration file
+
+Create `docs-i18n.config.ts` in your project root:
+
+```ts
+import { defineConfig } from 'docs-i18n';
+
+export default defineConfig({
+  projects: {
+    mydocs: {
+      sources: {
+        latest: 'content/docs',
+      },
+    },
+  },
+  languages: ['zh-hans', 'ja', 'es'],
+  llm: {
+    provider: 'openrouter',
+    model: 'deepseek/deepseek-chat-v3-0324:free',
+    apiKey: process.env.OPENROUTER_API_KEY,
+  },
+});
+```
+
+The `projects` object maps project names to their source directories. Each project can have multiple versions (e.g., `latest`, `v1`, `v2`). The `languages` array lists the target language codes to translate into.
+
+### 2. Scan your source files
+
+Before translating, scan your English source files to build the source index:
+
+```bash
+npx docs-i18n rescan
+```
+
+This parses every markdown/MDX file in your source directories, extracts translatable nodes, and stores them in the SQLite cache.
+
+### 3. Check translation status
+
+```bash
+npx docs-i18n status
+```
+
+This displays a progress bar for each language showing how many nodes have been translated.
+
+### 4. Run your first translation
+
+```bash
+npx docs-i18n translate --lang zh-hans
+```
+
+This sends untranslated nodes to your configured LLM, receives translations, and stores them in the cache. Only new or changed content is translated -- cached translations are reused.
+
+### 5. Assemble output files
+
+```bash
+npx docs-i18n assemble
+```
+
+This produces translated markdown files by combining your English sources with cached translations. Output is written to `.cache/content/<version>/<lang>/` by default.
+
+## Minimal Working Example
+
+Given this project structure:
+
+```
+my-docs/
+  content/
+    docs/
+      getting-started.md
+      api-reference.md
+  docs-i18n.config.ts
+  package.json
+```
+
+With this config:
+
+```ts
+import { defineConfig } from 'docs-i18n';
+
+export default defineConfig({
+  projects: {
+    docs: {
+      sources: { latest: 'content/docs' },
+    },
+  },
+  languages: ['zh-hans'],
+  llm: {
+    provider: 'openrouter',
+    apiKey: process.env.OPENROUTER_API_KEY,
+    model: 'deepseek/deepseek-chat-v3-0324:free',
+    contextLength: 32768,
+    maxTokens: 16384,
+  },
+  context: 'MyProject is a TypeScript library for building web apps.',
+});
+```
+
+Run these commands:
+
+```bash
+npx docs-i18n rescan
+npx docs-i18n translate --lang zh-hans
+npx docs-i18n assemble --lang zh-hans
+```
+
+The translated files will appear at:
+
+```
+.cache/content/latest/zh-hans/getting-started.md
+.cache/content/latest/zh-hans/api-reference.md
+```
+
+## Adding npm scripts
+
+Add these convenience scripts to your `package.json`:
+
+```json
+{
+  "scripts": {
+    "i18n": "docs-i18n",
+    "i18n:status": "docs-i18n status",
+    "i18n:translate": "docs-i18n translate",
+    "i18n:assemble": "docs-i18n assemble",
+    "i18n:rescan": "docs-i18n rescan",
+    "i18n:admin": "docs-i18n admin"
+  }
+}
+```