vite-plugin-ai-auto-i18n 0.1.0 → 0.2.0

package/README.md

@@ -1,8 +1,8 @@
-# vite-plugin-autoi18n — MVP sketch
+# 🌍 vite-plugin-autoi18n — MVP sketch 🚀
 
 Implicit build-time localization for JSX: author English directly, let the compiler extract user-facing copy, persist message identities, and optionally translate through an AI provider.
 
-## Current MVP scope
+## 🧩 Current MVP scope
 
 - Extracts JSX text nodes and static string JSX attributes.
 - Uses tag/attribute/semantic-attribute/handler context to form fingerprints; component names and file paths are excluded so ordinary moves and renames do not automatically invalidate IDs.
@@ -11,37 +11,70 @@ Implicit build-time localization for JSX: author English directly, let the compi
 - Supports a pluggable translation provider; includes an OpenAI-compatible implementation.
 - Writes locale JSON catalogues and keeps source English as fallback.
 - Two delivery modes: `build` (one compiled bundle per locale) and `runtime` (one bundle, locales loaded on demand through `virtual:autoi18n`).
-- Re-translates a message when its source text changes, tracked in `.autoi18n/locales/<locale>.sources.json`.
+- Re-translates a message when its source text changes, tracked in the `sources` map of `.autoi18n/locales/<locale>.json`.
 - Includes a deliberately varied document-dashboard example.
 
-## Install / try
+## 📦 Installation
 
-This repository is a plugin prototype. Add dependencies from `package.json`, then configure it in Vite:
+### 1️⃣ Install the package
+
+In your React + Vite project:
+
+```sh
+npm install -D vite-plugin-ai-auto-i18n
+```
+
+Works the same with `pnpm add -D vite-plugin-ai-auto-i18n` or `yarn add -D vite-plugin-ai-auto-i18n`. It requires Vite 5 or newer (a peer dependency). For React, also keep `@vitejs/plugin-react` in your Vite config.
+
+### 2️⃣ Add your API key
+
+Put the key in `.env.local` and keep that file out of git:
+
+```sh
+OPENAI_API_KEY=your-key-here
+```
+
+### 3️⃣ Configure Vite
+
+Add the plugin to `vite.config.ts`. `loadEnv` reads `.env.local` for the config file:
 
 ```ts
-import autoI18n, { OpenAICompatibleProvider } from "vite-plugin-autoi18n";
-
-export default defineConfig({
-  plugins: [
-    autoI18n({
-      locales: ["en", "de", "fr"],
-      sourceLocale: "en",
-      provider: new OpenAICompatibleProvider({
-        apiKey: process.env.OPENAI_API_KEY!,
-        model: "gpt-4.1-mini"
-      })
-    })
-  ]
+import { defineConfig, loadEnv } from "vite";
+import react from "@vitejs/plugin-react";
+import autoI18n, { OpenAICompatibleProvider } from "vite-plugin-ai-auto-i18n";
+
+export default defineConfig(({ mode }) => {
+  const env = loadEnv(mode, process.cwd(), "");
+
+  return {
+    plugins: [
+      autoI18n({
+        locales: ["en", "de", "fr"],
+        sourceLocale: "en",
+        provider: new OpenAICompatibleProvider({
+          apiKey: env.OPENAI_API_KEY!,
+          model: "gpt-4.1-mini"
+        })
+      }),
+      react() // in runtime delivery, keep autoI18n before the React plugin
+    ]
+  };
 });
 ```
 
+### 4️⃣ Commit the generated files
+
+The build writes `.autoi18n/` (registry and catalogues). Commit it so existing translations are reused instead of requested again. ✅
+
 Then write ordinary JSX:
 
 ```tsx
 <button title="Save the current document">Save</button>
 ```
 
-To try the bundled example (a React document dashboard under `example/`):
+### 🧪 Try the bundled example
+
+A React document dashboard under `example/`:
 
 ```sh
 npm install
@@ -49,25 +82,31 @@ npm run example:dev     # dev server
 npm run example:build   # production build into example/dist
 ```
 
-Run Vite. The plugin writes `.autoi18n/registry.json`, `.autoi18n/locales/<locale>.json` and `.autoi18n/locales/<locale>.sources.json`. The sidecar records the source text each translation was made from; if the source changes, that message is re-translated on the next build.
+Run Vite. The plugin writes `.autoi18n/registry.json` and `.autoi18n/locales/<locale>.json`. Each locale file has two parts: `messages` (the translations) and `sources` (the source text each translation was made from). If the source changes, that message is re-translated on the next build.
+
+Upgrading from an earlier version: the old flat locale files and `*.sources.json` sidecars are not read. Every message is re-translated once on the first build, and the sidecars are deleted.
+
+If a translation batch fails, the other batches are still stored. The failed messages fall back to the source text until the next build. If every batch for a locale fails, that locale keeps its previous file and the build reports the error after writing the other locales.
+
+🔑 Translations require a provider. With `failOnMissingProvider: true` the build fails when a locale is missing translations and no provider is configured, instead of silently shipping English.
 
-Translations require a provider. With `failOnMissingProvider: true` the build fails when a locale is missing translations and no provider is configured, instead of silently shipping English.
+`OpenAICompatibleProvider` sends messages in batches (`batchSize`, default 40), runs up to `concurrency` batches at once (default 4), aborts a request after `timeoutMs` (default 60000), and retries network errors, timeouts, 408/409/429 and 5xx responses or malformed output up to `maxRetries` times (default 2) with exponential backoff from `retryDelayMs` (default 500). Other 4xx errors fail immediately. If the provider omits some messages, the rest are kept and the missing ones stay in the source language until the next build retries them; the build logs how many were left untranslated.
 
-`OpenAICompatibleProvider` sends messages in batches (`batchSize`, default 40), aborts a request after `timeoutMs` (default 60000), and retries network errors, timeouts, 408/409/429 and 5xx responses or malformed output up to `maxRetries` times (default 2) with exponential backoff from `retryDelayMs` (default 500). Other 4xx errors fail immediately. If the provider omits some messages, the rest are kept and the missing ones stay in the source language until the next build retries them; the build logs how many were left untranslated.
+The plugin translates up to `localeConcurrency` locales at once (default 2). Total parallel requests are roughly `localeConcurrency × concurrency`, so lower one or both if your provider rate-limits your account.
 
 Ambiguous identity matches (duplicate identical strings whose count changed) are logged as warnings. Set `onAmbiguous: "error"` to fail the build instead.
 
 Messages removed from the source are kept as orphans in `registry.json` so their IDs can be reused if the text returns. Orphans not seen for `orphanRetentionMinutes` minutes (default `43200`, i.e. 30 days) are deleted on the next build, and the build logs how many were removed. Use `0` to delete them immediately or `Infinity` to keep them forever. Once deleted, a returning message gets a new ID and is translated again.
 
-To prune without running a build, use the CLI. In this repository, `npm run prune` builds and then runs it on the example app's `example/.autoi18n/` with `--retention-minutes 0`, so every orphan is removed. Pass options after `--` to override, for example `npm run prune -- --directory .autoi18n --retention-minutes 1440 --debug`. Options: `--directory` (default `.autoi18n`, relative to the current directory), `--retention-minutes` (default `43200`, i.e. 30 days, the plugin default) and `--debug` (prints each removed entry with its source text and last-seen time). The CLI also removes any key from `locales/*.json` and `locales/*.sources.json` that is no longer an active registry entry, so the catalogues match the registry. If no registry exists at that path, the CLI says so and changes nothing. Installed packages can run `node node_modules/vite-plugin-autoi18n/dist/cli/prune.js` with the same options.
+🧹 To prune without running a build, use the CLI. In this repository, `npm run prune` builds and then runs it on the example app's `example/.autoi18n/` with `--retention-minutes 0`, so every orphan is removed. Pass options after `--` to override, for example `npm run prune -- --directory .autoi18n --retention-minutes 1440 --debug`. Options: `--directory` (default `.autoi18n`, relative to the current directory), `--retention-minutes` (default `43200`, i.e. 30 days, the plugin default) and `--debug` (prints each removed entry with its source text and last-seen time). The CLI also removes any key from the `messages` and `sources` maps of `locales/*.json` that is no longer an active registry entry, so the catalogues match the registry. If no registry exists at that path, the CLI says so and changes nothing. Installed packages can run `node node_modules/vite-plugin-ai-auto-i18n/dist/cli/prune.js` with the same options.
 
-Set `debug: true` (default `false`) to print every translation exchange to the console: each message sent to the provider (`sent`) and each translation received back (`received`), prefixed with `[autoi18n debug]` and an ISO timestamp.
+🐞 Set `debug: true` (default `false`) to print every translation exchange to the console: each message sent to the provider (`sent`) and each translation received back (`received`), prefixed with `[autoi18n debug]` and an ISO timestamp.
 
-To keep copy out of automatic translation, add `translate="no"` to the element (for example `<h2 translate="no">Sample code</h2>`). This is the standard HTML marker, and it also covers the element's descendants and attributes such as `title`.
+🙈 To keep copy out of automatic translation, add `translate="no"` to the element (for example `<h2 translate="no">Sample code</h2>`). This is the standard HTML marker, and it also covers the element's descendants and attributes such as `title`.
 
 Corrupt `registry.json` or catalogue files are renamed to `<file>.corrupt-<timestamp>` and a warning is logged; the plugin continues with an empty state. Writes are atomic, so an interrupted build does not leave a truncated file.
 
-### Delivery modes
+### 🚚 Delivery modes
 
 - `delivery: "build"` (default): compiles one locale per build. Use `AUTOI18N_LOCALE=de vite build` (on Windows use `cross-env AUTOI18N_LOCALE=de vite build`) or `locale: "de"` in plugin options, typically in a CI build matrix. When a locale is set explicitly, only that locale is translated during the build. Without one, all configured locales are translated. An unknown locale fails the build.
 - `delivery: "runtime"`: one bundle for all locales. Text is wrapped in `t(id, source)` calls and catalogues are fetched on demand. Import the API from the virtual module:
@@ -78,13 +117,82 @@ import { locales, resolveSupported, setLocale, getLocale, subscribe, t } from "v
 setLocale(resolveSupported(navigator.languages)); // call before first render
 ```
 
-`createLocaleLoader` and `resolveLocale` are available from the `vite-plugin-autoi18n/runtime` subpath, not the main entry.
+`createLocaleLoader` and `resolveLocale` are available from the `vite-plugin-ai-auto-i18n/runtime` subpath, not the main entry.
 
 `subscribe(listener)` returns an unsubscribe function. `getVersion()` changes whenever the locale or the catalogues change, so `useSyncExternalStore(subscribe, getVersion, getVersion)` re-renders components that read `t()` during render. Components wrapped in `memo` must subscribe themselves, or they keep the old text until their props change. List `autoI18n()` before `@vitejs/plugin-react` in `plugins`. The React plugin rewrites JSX in dev before other pre-transforms run, so the plugin must see the original JSX. Add a type declaration for `virtual:autoi18n` (see [example/src/autoi18n-virtual.d.ts](/home/bogdan/git/vite-plugin-ai-i18n/example/src/autoi18n-virtual.d.ts)). The example app uses this mode with a locale switcher.
 
 In dev, saving a source file re-indexes the project, translates only new or changed messages and pushes the updated catalogues to the browser over an `autoi18n:update` event. Files added or removed are indexed as well. The page is not reloaded, so new JSX text appears once its translation is ready.
 
-## Important prototype limitations
+## ⚙️ Configuration reference
+
+### Plugin options (`autoI18n({...})`)
+
+| Option | Type | Default | Description |
+|---|---|---|---|
+| `locales` | `string[]` | _required_ | Locales the app ships, for example `["en", "de", "fr"]`. The source locale is never translated, so including it is harmless. |
+| `sourceLocale` | `string` | `"en"` | Language the JSX is written in. Used as the fallback when a translation is missing. |
+| `delivery` | `"build" \| "runtime"` | `"build"` | `build` compiles one locale per build. `runtime` ships every catalogue and switches locale in the browser with `setLocale()` from `virtual:autoi18n`. |
+| `locale` | `string` | `sourceLocale` | Locale to compile in `build` delivery. Also read from the `AUTOI18N_LOCALE` environment variable, which is handy for CI matrix builds. In `build` delivery during a non-dev build, only this locale is translated. |
+| `include` | `string[]` | `["src/**/*.{tsx,jsx,ts,js}"]` | Glob patterns of source files to scan, relative to the Vite root. |
+| `exclude` | `string[]` | `["**/*.test.*", "**/node_modules/**"]` | Glob patterns of files to skip, relative to the Vite root. |
+| `directory` | `string` | `".autoi18n"` | Folder for `registry.json` and `locales/`, relative to the Vite root. |
+| `provider` | `TranslationProvider` | `undefined` | Translates missing and changed messages. Without one, untranslated messages fall back to the source text. |
+| `failOnMissingProvider` | `boolean` | `false` | When `true`, the build fails if a locale has untranslated messages and no `provider` is set. |
+| `onAmbiguous` | `"warn" \| "error"` | `"warn"` | What to do when duplicated identical texts change their occurrence count, so their IDs may have moved. `"error"` fails the build. |
+| `orphanRetentionMinutes` | `number` | `43200` (30 days) | Minutes an orphaned message is kept before it is deleted. `0` deletes on the next build; `Infinity` keeps forever. |
+| `localeConcurrency` | `number` | `2` | Locales translated at the same time. Values below 1 are raised to 1, and fractions are rounded down. |
+| `debug` | `boolean` | `false` | Logs each message sent to the provider, each translation received, and each removed orphan, with timestamps. |
+
+### `OpenAICompatibleProvider` options
+
+Create the provider with `new OpenAICompatibleProvider(options)`.
+
+| Option | Type | Default | Description |
+|---|---|---|---|
+| `apiKey` | `string` | _required_ | Bearer token for the endpoint. Read it from an environment variable and do not commit it. |
+| `model` | `string` | _required_ | Model name sent in each request, for example `"gpt-4.1-mini"`. |
+| `baseURL` | `string` | `"https://api.openai.com/v1"` | Base URL of any OpenAI-compatible API. Requests go to `${baseURL}/chat/completions`. |
+| `batchSize` | `number` | `40` | Messages per request. Smaller batches keep responses within output limits. |
+| `concurrency` | `number` | `4` | Batches sent in parallel for one locale. |
+| `timeoutMs` | `number` | `60000` | Milliseconds before a single request is aborted. |
+| `maxRetries` | `number` | `2` | Extra attempts for network errors, timeouts, 408/409/429/5xx responses and malformed output. |
+| `retryDelayMs` | `number` | `500` | Delay before the first retry. It doubles on each attempt. |
+
+### Custom provider (`TranslationProvider`)
+
+Any object with this shape can be passed as `provider`:
+
+```ts
+interface TranslationProvider {
+  translate(input: {
+    sourceLocale: string;
+    targetLocale: string;
+    messages: TranslationRequest[];
+  }): Promise<Record<string, string>>; // message ID -> translated text
+}
+
+interface TranslationRequest {
+  id: string;             // opaque ID to echo back in the result
+  source: string;         // English text
+  context: string;        // for example "Toolbar > Save button"
+  kind: "text" | "attribute" | "mixed"; // text node, static attribute, or both in one message
+  placeholders?: string[]; // {name} placeholders that must survive translation
+}
+```
+
+Return only the IDs you translated. Omitted IDs stay in the source language and are requested again on the next build. If your provider can only translate some batches, throw `PartialTranslationError` (exported from the package) with the translations you have and the number of failed messages, so the successful ones are still stored. Any other error fails the whole locale, which keeps its previous file.
+
+### Environment variables
+
+| Variable | Effect |
+|---|---|
+| `AUTOI18N_LOCALE` | Same as the `locale` option. The option wins when both are set. |
+
+### Prune CLI
+
+The `prune` CLI options (`--directory`, `--retention-minutes`, `--debug`) are described in the orphan paragraph of [Try the bundled example](#-try-the-bundled-example).
+
+## ⚠️ Important prototype limitations
 
 This is a compiler MVP, not production-ready localization yet:
 
@@ -96,7 +204,7 @@ This is a compiler MVP, not production-ready localization yet:
 6. The example uses a React plugin in its Vite config; install `@vitejs/plugin-react` in the consuming example.
 7. The initial fingerprint is a heuristic. It excludes file and line position, but component names and nearby semantic signals can still change during refactors.
 
-## Design rules
+## 🧭 Design rules
 
 - IDs are opaque and persistent; never expose semantic fingerprints as runtime keys.
 - AI suggestions never silently overwrite an existing translation.

package/dist/{chunk-ASK7BKZ3.js → chunk-LLL7ZSKW.js}

@@ -2,9 +2,11 @@
 var debugLog = (message) => console.log(`[autoi18n debug] ${(/* @__PURE__ */ new Date()).toISOString()} ${message}`);
 
 // src/io.ts
-import { randomUUID } from "crypto";
+import { createHash, randomUUID } from "crypto";
 import { mkdir, readFile, rename, writeFile } from "fs/promises";
 import path from "path";
+var toJson = (value) => JSON.stringify(value, null, 2) + "\n";
+var hashContent = (text) => createHash("sha256").update(text).digest("hex");
 async function readJsonFile(file, fallback, warn) {
   let text;
   try {
@@ -30,6 +32,28 @@ async function writeFileAtomic(file, content) {
   await rename(temp, file);
 }
 
+// src/catalog.ts
+import path2 from "path";
+var emptyLocaleFile = () => ({ messages: {}, sources: {} });
+function localeFilePath(outDir, locale) {
+  return path2.join(outDir, `${locale}.json`);
+}
+async function readLocaleFile(file, warn) {
+  const parsed = await readJsonFile(file, null, warn);
+  if (!parsed || typeof parsed.messages !== "object" || typeof parsed.sources !== "object") return emptyLocaleFile();
+  return { messages: parsed.messages, sources: parsed.sources };
+}
+var writeLocaleFile = (file, content) => writeFileAtomic(file, toJson(content));
+function pruneCatalog(catalog, keep) {
+  const kept = {};
+  const removed = [];
+  for (const [id, text] of Object.entries(catalog)) {
+    if (keep.has(id)) kept[id] = text;
+    else removed.push(id);
+  }
+  return { catalog: kept, removed };
+}
+
 // src/identity/registry.ts
 import { randomUUID as randomUUID2 } from "crypto";
 var emptyRegistry = () => ({ version: 1, entries: {} });
@@ -63,6 +87,7 @@ function reconcile(registry, messages) {
     list.push(message);
     incoming.set(message.fingerprint, list);
   }
+  const assigned = /* @__PURE__ */ new Map();
   const seen = /* @__PURE__ */ new Set();
   const ambiguous = [];
   for (const [fingerprint, current] of incoming) {
@@ -76,7 +101,7 @@ function reconcile(registry, messages) {
       const message = current[i];
       const candidate = previous[i];
       if (candidate) {
-        message.id = candidate.id;
+        assigned.set(message, candidate.id);
         candidate.status = "active";
         candidate.lastSeen = now;
         candidate.source = message.source;
@@ -84,7 +109,7 @@ function reconcile(registry, messages) {
         seen.add(candidate.id);
       } else {
         const id = `msg_${randomUUID2().replaceAll("-", "").slice(0, 16)}`;
-        message.id = id;
+        assigned.set(message, id);
         registry.entries[id] = {
           id,
           source: message.source,
@@ -104,7 +129,12 @@ function reconcile(registry, messages) {
     }
   }
   for (const entry of Object.values(registry.entries)) if (!seen.has(entry.id)) entry.status = "orphaned";
-  return { registry, ambiguous };
+  const resolved = messages.map((message) => {
+    const id = assigned.get(message);
+    if (id === void 0) throw new Error(`autoi18n: no message ID was assigned to ${message.file}:${message.start}`);
+    return { ...message, id };
+  });
+  return { registry, ambiguous, messages: resolved };
 }
 var MINUTE_MS = 60 * 1e3;
 var DEFAULT_ORPHAN_RETENTION_MINUTES = 30 * 24 * 60;
@@ -120,13 +150,17 @@ function pruneOrphans(registry, retentionMinutes, now = Date.now()) {
   return removed;
 }
 async function saveRegistry(file, registry) {
-  await writeFileAtomic(file, JSON.stringify(registry, null, 2) + "\n");
+  await writeFileAtomic(file, toJson(registry));
 }
 
 export {
   debugLog,
-  readJsonFile,
-  writeFileAtomic,
+  hashContent,
+  emptyLocaleFile,
+  localeFilePath,
+  readLocaleFile,
+  writeLocaleFile,
+  pruneCatalog,
   loadRegistry,
   reconcile,
   DEFAULT_ORPHAN_RETENTION_MINUTES,

package/dist/chunk-ZDZ3YVET.js

@@ -0,0 +1,38 @@
+// src/runtime/loader.ts
+function createLocaleLoader(loaders, fallback = "en") {
+  const cache = /* @__PURE__ */ new Map();
+  function loadLocale(locale) {
+    const selected = loaders[locale] ? locale : fallback;
+    const cached = cache.get(selected);
+    if (cached) return cached;
+    const loader = loaders[selected];
+    if (!loader) return Promise.reject(new Error(`No catalogue loader for locale "${selected}"`));
+    const pending = loader().then((module) => "default" in module ? module.default : module);
+    cache.set(selected, pending);
+    pending.catch(() => {
+      if (cache.get(selected) === pending) cache.delete(selected);
+    });
+    return pending;
+  }
+  loadLocale.prime = (locale, catalog) => {
+    cache.set(locale, Promise.resolve(catalog));
+  };
+  return loadLocale;
+}
+
+// src/runtime/resolve.ts
+function resolveLocale(preferred, supported, fallback = "en") {
+  for (const candidate of preferred) {
+    const exact = supported.find((x) => x.toLowerCase() === candidate.toLowerCase());
+    if (exact) return exact;
+    const base = candidate.split("-")[0].toLowerCase();
+    const language = supported.find((x) => x.split("-")[0].toLowerCase() === base);
+    if (language) return language;
+  }
+  return fallback;
+}
+
+export {
+  createLocaleLoader,
+  resolveLocale
+};

package/dist/cli/prune.js

@@ -2,27 +2,29 @@ import {
   DEFAULT_ORPHAN_RETENTION_MINUTES,
   debugLog,
   loadRegistry,
+  pruneCatalog,
   pruneOrphans,
-  readJsonFile,
+  readLocaleFile,
   saveRegistry,
-  writeFileAtomic
-} from "../chunk-ASK7BKZ3.js";
+  writeLocaleFile
+} from "../chunk-LLL7ZSKW.js";
 
 // src/cli/prune.ts
 import { existsSync } from "fs";
 import { readdir } from "fs/promises";
 import path from "path";
-async function pruneCatalogs(localesDir, activeIds) {
+async function pruneLocaleFiles(localesDir, activeIds) {
   if (!existsSync(localesDir)) return [];
   const results = [];
   for (const name of await readdir(localesDir)) {
     if (!name.endsWith(".json")) continue;
     const file = path.join(localesDir, name);
-    const catalog = await readJsonFile(file, {}, (message) => console.warn(message));
-    const removed = Object.keys(catalog).filter((id) => !activeIds.has(id));
+    const current = await readLocaleFile(file, (message) => console.warn(message));
+    const messages = pruneCatalog(current.messages, activeIds);
+    const sources = pruneCatalog(current.sources, activeIds);
+    const removed = [.../* @__PURE__ */ new Set([...messages.removed, ...sources.removed])];
     if (!removed.length) continue;
-    for (const id of removed) delete catalog[id];
-    await writeFileAtomic(file, JSON.stringify(catalog, null, 2) + "\n");
+    await writeLocaleFile(file, { messages: messages.catalog, sources: sources.catalog });
     results.push({ file, removed });
   }
   return results;
@@ -56,7 +58,7 @@ async function main() {
   const removed = pruneOrphans(registry, args.retentionMinutes);
   if (removed.length) await saveRegistry(registryFile, registry);
   const activeIds = new Set(Object.values(registry.entries).filter((entry) => entry.status === "active").map((entry) => entry.id));
-  const catalogs = await pruneCatalogs(path.join(path.dirname(registryFile), "locales"), activeIds);
+  const catalogs = await pruneLocaleFiles(path.join(path.dirname(registryFile), "locales"), activeIds);
   if (args.debug) {
     for (const entry of removed) debugLog(`removed ${entry.id} ${JSON.stringify(entry.source)} (last seen ${entry.lastSeen})`);
     for (const { file, removed: ids } of catalogs) {