docs-i18n 0.8.0 → 0.8.2

package/template/content/docs-i18n/en/deployment.md

@@ -0,0 +1,209 @@
+---
+title: Deployment
+description: Using docs-i18n in CI/CD pipelines, deploying the admin dashboard, and runtime translation with Cloudflare D1.
+---
+
+# Deployment
+
+## CI/CD with GitHub Actions
+
+docs-i18n works well in CI/CD pipelines. Since translations are cached in SQLite, you only pay for API calls on new or changed content.
+
+### Basic translation workflow
+
+```yaml
+name: Translate Docs
+on:
+  push:
+    branches: [main]
+    paths:
+      - 'content/**'
+
+jobs:
+  translate:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 20
+
+      - run: npm ci
+
+      # Restore the SQLite cache from a previous run
+      - uses: actions/cache@v4
+        with:
+          path: .cache
+          key: i18n-cache-${{ hashFiles('content/**') }}
+          restore-keys: |
+            i18n-cache-
+
+      # Rescan source files to detect changes
+      - run: npx docs-i18n rescan
+
+      # Translate all configured languages
+      - run: npx docs-i18n translate --lang zh-hans
+        env:
+          OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
+
+      - run: npx docs-i18n translate --lang ja
+        env:
+          OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
+
+      # Assemble output files
+      - run: npx docs-i18n assemble
+
+      # Commit translated files (or use them in a build step)
+      - uses: stefanzweifel/git-auto-commit-action@v5
+        with:
+          commit_message: 'chore: update translations'
+          file_pattern: '.cache/content/**'
+```
+
+### Tips for CI/CD
+
+- **Cache the SQLite database.** The `.cache/translations.db` file contains all cached translations. Restoring it between runs avoids re-translating unchanged content. Use `actions/cache` with a key based on your content files.
+- **Run `rescan` before `translate`.** This detects new, changed, and deleted source files, and cleans up orphaned translations.
+- **Use `--dry-run` in PR checks.** Add a CI step that runs `docs-i18n translate --lang zh-hans --dry-run` to report how many nodes need translation without spending API credits.
+- **Limit concurrency.** In CI, you may want `--concurrency 1` to avoid rate limiting from your LLM provider.
+- **Set `--max` for budgeting.** Use `--max 10` to cap the number of API calls per run if you want to translate incrementally across multiple CI runs.
+
+### PR status check
+
+```yaml
+name: Translation Status
+on:
+  pull_request:
+    paths:
+      - 'content/**'
+
+jobs:
+  check:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 20
+      - run: npm ci
+
+      - uses: actions/cache@v4
+        with:
+          path: .cache
+          key: i18n-cache-${{ hashFiles('content/**') }}
+          restore-keys: |
+            i18n-cache-
+
+      - run: npx docs-i18n rescan
+      - run: npx docs-i18n status
+      - run: npx docs-i18n translate --lang zh-hans --dry-run
+```
+
+## Admin Dashboard Deployment
+
+The admin dashboard is a TanStack Start application that runs locally. It is designed for development use, not production deployment.
+
+To start it:
+
+```bash
+npx docs-i18n admin
+# or
+npx docs-i18n admin --port 4000
+```
+
+The dashboard starts a Vite dev server pointed at the `src/admin` directory within the docs-i18n package. It reads your `docs-i18n.config.ts` to discover projects, versions, and languages.
+
+**Prerequisites:**
+
+Your project must have `vite` and `@vitejs/plugin-react` installed as dev dependencies:
+
+```bash
+npm install -D vite @vitejs/plugin-react
+```
+
+## Runtime Translation with D1
+
+For SSR sites that fetch markdown at runtime (e.g., TanStack Start on Cloudflare Workers), docs-i18n provides a lightweight runtime translator that queries Cloudflare D1 for cached translations.
+
+### How it works
+
+The `createTranslator` function from `docs-i18n/serve` creates a translator instance. When `translate()` is called:
+
+1. The markdown is parsed into AST nodes using the same parser as the CLI.
+2. Translatable nodes' MD5 keys are collected.
+3. A batch query is sent to D1 to look up all translations for the given language.
+4. The content is reassembled: translated nodes use the D1 cache value, untranslated nodes fall back to the original English text.
+
+### Setup
+
+#### 1. Export translations to D1
+
+First, translate your content using the CLI as normal. Then export the SQLite cache to D1. You can use Wrangler to create a D1 database and import the translations:
+
+```bash
+# Create a D1 database
+wrangler d1 create docs-translations
+
+# Export the translations table from your local SQLite
+sqlite3 .cache/translations.db ".dump translations" > translations.sql
+
+# Import into D1
+wrangler d1 execute docs-translations --file=translations.sql
+```
+
+Make sure the D1 database has the same `translations` table schema:
+
+```sql
+CREATE TABLE IF NOT EXISTS translations (
+  lang       TEXT NOT NULL,
+  key        TEXT NOT NULL,
+  value      TEXT NOT NULL,
+  created_at INTEGER NOT NULL DEFAULT (unixepoch()),
+  updated_at INTEGER NOT NULL DEFAULT (unixepoch()),
+  PRIMARY KEY (lang, key)
+);
+```
+
+#### 2. Use the translator in your Worker
+
+```ts
+import { createTranslator } from 'docs-i18n/serve';
+
+const translator = createTranslator();
+
+// In your request handler (e.g., TanStack Start server function):
+export async function getTranslatedDoc(
+  repo: string,
+  branch: string,
+  filePath: string,
+  lang: string,
+  env: { DB: D1Database },
+) {
+  // Fetch the English markdown from GitHub
+  const enMarkdown = await fetchFromGitHub(repo, branch, filePath);
+
+  // If the requested language is English, return as-is
+  if (lang === 'en') return enMarkdown;
+
+  // Translate using D1 cache
+  const translated = await translator.translate(enMarkdown, lang, env.DB);
+  return translated;
+}
+```
+
+#### 3. Bind D1 in wrangler.toml
+
+```toml
+[[d1_databases]]
+binding = "DB"
+database_name = "docs-translations"
+database_id = "your-database-id"
+```
+
+### Performance considerations
+
+- The translator uses a single batch query to D1, so latency scales with the number of translatable nodes (typically under 50ms for a standard documentation page).
+- English content (`lang === 'en'`) short-circuits immediately without parsing or querying.
+- The parser runs on every request since the English source is fetched at runtime. For heavy traffic, consider caching the translated output at the edge.
+- Untranslated nodes gracefully fall back to English, so partial translations work without errors.

package/template/content/docs-i18n/en/getting-started.md

@@ -0,0 +1,168 @@
+---
+title: Getting Started
+description: Install docs-i18n and translate your first documentation file in minutes.
+---
+
+# Getting Started
+
+## What is docs-i18n?
+
+docs-i18n is a universal documentation translation engine. It parses markdown and MDX files into translatable AST nodes, translates them via LLM, caches results in SQLite, and assembles translated output files. It is framework-agnostic and works with any documentation site built on Next.js, Astro, TanStack Start, or similar tools.
+
+Key characteristics:
+
+- **Incremental** -- only changed or new content is sent to the LLM.
+- **AST-based** -- uses remark to parse markdown into nodes, producing stable MD5 keys for deduplication.
+- **SQLite-backed** -- concurrent-safe cache with WAL mode. No manual save steps.
+- **Multi-project** -- supports multiple documentation projects and versions in a single config.
+- **Admin dashboard** -- web UI for monitoring progress, managing jobs, and previewing translations.
+- **Runtime serve** -- optional D1-compatible translator for SSR sites on Cloudflare Workers.
+
+## Requirements
+
+- Node.js 20+ or Bun
+- An API key for an LLM provider (OpenRouter, OpenAI, or Anthropic)
+- For the admin dashboard: `vite` and `@vitejs/plugin-react` as dev dependencies
+
+## Installation
+
+```bash
+npm install docs-i18n
+```
+
+Or with Bun:
+
+```bash
+bun add docs-i18n
+```
+
+## Quick Start
+
+### 1. Create a configuration file
+
+Create `docs-i18n.config.ts` in your project root:
+
+```ts
+import { defineConfig } from 'docs-i18n';
+
+export default defineConfig({
+  projects: {
+    mydocs: {
+      sources: {
+        latest: 'content/docs',
+      },
+    },
+  },
+  languages: ['zh-hans', 'ja', 'es'],
+  llm: {
+    provider: 'openrouter',
+    model: 'deepseek/deepseek-chat-v3-0324:free',
+    apiKey: process.env.OPENROUTER_API_KEY,
+  },
+});
+```
+
+The `projects` object maps project names to their source directories. Each project can have multiple versions (e.g., `latest`, `v1`, `v2`). The `languages` array lists the target language codes to translate into.
+
+### 2. Scan your source files
+
+Before translating, scan your English source files to build the source index:
+
+```bash
+npx docs-i18n rescan
+```
+
+This parses every markdown/MDX file in your source directories, extracts translatable nodes, and stores them in the SQLite cache.
+
+### 3. Check translation status
+
+```bash
+npx docs-i18n status
+```
+
+This displays a progress bar for each language showing how many nodes have been translated.
+
+### 4. Run your first translation
+
+```bash
+npx docs-i18n translate --lang zh-hans
+```
+
+This sends untranslated nodes to your configured LLM, receives translations, and stores them in the cache. Only new or changed content is translated -- cached translations are reused.
+
+### 5. Assemble output files
+
+```bash
+npx docs-i18n assemble
+```
+
+This produces translated markdown files by combining your English sources with cached translations. Output is written to `.cache/content/<version>/<lang>/` by default.
+
+## Minimal Working Example
+
+Given this project structure:
+
+```
+my-docs/
+  content/
+    docs/
+      getting-started.md
+      api-reference.md
+  docs-i18n.config.ts
+  package.json
+```
+
+With this config:
+
+```ts
+import { defineConfig } from 'docs-i18n';
+
+export default defineConfig({
+  projects: {
+    docs: {
+      sources: { latest: 'content/docs' },
+    },
+  },
+  languages: ['zh-hans'],
+  llm: {
+    provider: 'openrouter',
+    apiKey: process.env.OPENROUTER_API_KEY,
+    model: 'deepseek/deepseek-chat-v3-0324:free',
+    contextLength: 32768,
+    maxTokens: 16384,
+  },
+  context: 'MyProject is a TypeScript library for building web apps.',
+});
+```
+
+Run these commands:
+
+```bash
+npx docs-i18n rescan
+npx docs-i18n translate --lang zh-hans
+npx docs-i18n assemble --lang zh-hans
+```
+
+The translated files will appear at:
+
+```
+.cache/content/latest/zh-hans/getting-started.md
+.cache/content/latest/zh-hans/api-reference.md
+```
+
+## Adding npm scripts
+
+Add these convenience scripts to your `package.json`:
+
+```json
+{
+  "scripts": {
+    "i18n": "docs-i18n",
+    "i18n:status": "docs-i18n status",
+    "i18n:translate": "docs-i18n translate",
+    "i18n:assemble": "docs-i18n assemble",
+    "i18n:rescan": "docs-i18n rescan",
+    "i18n:admin": "docs-i18n admin"
+  }
+}
+```

package/template/content/docs.config.json

@@ -0,0 +1,25 @@
+{
+  "sections": [
+    {
+      "label": "Getting Started",
+      "children": [
+        { "label": "Introduction", "to": "getting-started" }
+      ]
+    },
+    {
+      "label": "Usage",
+      "children": [
+        { "label": "CLI Commands", "to": "cli" },
+        { "label": "Configuration", "to": "configuration" },
+        { "label": "Admin Dashboard", "to": "admin" }
+      ]
+    },
+    {
+      "label": "Advanced",
+      "children": [
+        { "label": "Architecture", "to": "architecture" },
+        { "label": "Deployment", "to": "deployment" }
+      ]
+    }
+  ]
+}

package/template/content/en/admin.md

@@ -0,0 +1,151 @@
+---
+title: Admin Dashboard
+description: Web UI for monitoring translation progress, managing translation jobs, previewing files, and browsing LLM models.
+---
+
+# Admin Dashboard
+
+The docs-i18n admin dashboard is a web-based UI for managing your translations. It is built with TanStack Start and React, and runs as a local development server.
+
+## Starting the Dashboard
+
+```bash
+npx docs-i18n admin
+```
+
+The dashboard opens at `http://localhost:3456`. Use `--port` to change the port:
+
+```bash
+npx docs-i18n admin --port 4000
+```
+
+**Prerequisites:**
+
+Your project must have `vite` and `@vitejs/plugin-react` installed:
+
+```bash
+npm install -D vite @vitejs/plugin-react
+```
+
+The dashboard reads your `docs-i18n.config.ts` to discover projects, versions, and languages.
+
+## Features
+
+### Translation Overview
+
+The main dashboard page shows a grid of all versions and languages with translation progress. For each version/language pair, you see:
+
+- Total number of source nodes (EN content units).
+- Number of translated nodes.
+- Percentage complete.
+- Breakdown by section (e.g., `docs/`, `blog/`, `learn/`), showing file counts and node counts per section.
+
+English is always shown as 100% complete since it is the source language.
+
+The overview auto-scans source files on load. If source files have not been scanned yet, the dashboard triggers a scan automatically.
+
+### File Browser
+
+Clicking on a version/language cell opens a file-level coverage list. Each file shows:
+
+- File path (relative to the source directory).
+- Total translatable nodes in that file.
+- Number of translated nodes.
+
+Files are sorted by path. You can click on any file to open the block-level preview.
+
+### File Preview
+
+The file preview shows every block (AST node) in a file side by side:
+
+- **Source** -- the original English text.
+- **Translation** -- the cached translation for the selected language, or empty if not yet translated.
+
+Each block displays its type (heading, paragraph, list, blockquote, frontmatter, code, html) and MD5 key. Non-translatable blocks (code, pure HTML tags, gaps between nodes) are shown but clearly distinguished.
+
+### Cache Management
+
+From the file preview, you can delete individual cache entries. This is useful when a translation is incorrect and you want to re-translate a specific node. After deleting, run the translate command again to get a fresh translation for that key.
+
+The dashboard also supports rescanning source files for a specific version via the UI. This rebuilds the source index and cleans orphaned entries.
+
+### Translation Jobs
+
+The dashboard includes a job management system for running translations directly from the UI instead of the command line.
+
+#### Creating a Job
+
+Click the job creation button and configure:
+
+- **Language** -- target language code.
+- **Version** -- which version to translate.
+- **Project** -- optionally filter to a specific project.
+- **Model** -- LLM model to use (can be selected from the model browser).
+- **Model rotation** -- optionally provide multiple models to rotate through.
+- **Max chunks** -- limit the number of API call chunks.
+- **Concurrency** -- number of parallel API calls (default: 3).
+- **Files** -- optionally select specific files to translate.
+
+#### Job Status
+
+Running jobs show:
+
+- Status: `running`, `completed`, `failed`, or `cancelled`.
+- Start time and finish time.
+- Number of translated chunks and total chunks.
+- Current chunk being processed.
+- Live log output (last 20 lines displayed, up to 500 lines stored).
+
+You can cancel a running job, which sends SIGTERM to the translation process. Completed or failed jobs can be removed from the list.
+
+#### How Jobs Work
+
+Under the hood, the job manager spawns a child process running the `docs-i18n translate` CLI command with the configured options. It captures stdout and stderr, parses progress information from the output, and exposes it through the dashboard API. The child process inherits the API key from your config or environment variables.
+
+### Model Browser
+
+The dashboard includes an OpenRouter model browser that fetches the list of available models from the OpenRouter API. For each model, it displays:
+
+- Model ID and name.
+- Pricing (prompt and completion per million tokens).
+- Context length and maximum output tokens.
+- Whether the model supports JSON response format and tool use.
+- Provider name.
+- Whether the model is free.
+
+The model list is cached for 5 minutes. It only shows text-to-text models (filtered by architecture modality) and excludes models with negative pricing. Models are sorted by prompt price (cheapest first).
+
+This is useful for selecting a model when creating a translation job.
+
+### Open in Editor
+
+The dashboard can open source files in your local editor. It tries the following editors in order:
+
+1. The value of the `EDITOR_CMD` environment variable (if set).
+2. `code` (VS Code)
+3. `cursor` (Cursor)
+4. `zed` (Zed)
+
+If none are found, it falls back to the system default (`open` on macOS, `xdg-open` on Linux, `start` on Windows).
+
+## Architecture
+
+The admin dashboard uses:
+
+- **TanStack Start** -- Full-stack React framework with server functions.
+- **TanStack React Query** -- For data fetching and cache management.
+- **TanStack Router** -- For client-side routing.
+- **Vite** -- Dev server and build tool.
+- **Hono** -- HTTP server (used by TanStack Start internally).
+
+Server functions are defined in `src/admin/server/functions/` and handle:
+
+- `fetchStatus` / `fetchFileCoverage` / `fetchFileBlocks` -- Read translation status from SQLite.
+- `deleteCacheEntry` -- Delete a specific translation from the cache.
+- `rescanVersion` -- Rescan source files for a version.
+- `createJob` / `fetchJobs` / `fetchJob` / `deleteJob` -- Manage translation jobs.
+- `fetchModels` -- Fetch available models from OpenRouter.
+- `fetchVersion` / `fetchConfig` -- Get docs-i18n version and project root.
+- `openFile` -- Open a file in the local editor.
+
+The dashboard shares the same `TranslationCache` and `parseMdx` functions as the CLI, ensuring consistent behavior.