localewarden 0.1.1 → 0.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (56) hide show
  1. package/README.md +161 -12
  2. package/dist/budget.d.ts +25 -0
  3. package/dist/budget.js +74 -0
  4. package/dist/checks.d.ts +34 -6
  5. package/dist/checks.js +111 -21
  6. package/dist/cli.js +66 -10
  7. package/dist/config.d.ts +37 -1
  8. package/dist/config.js +118 -3
  9. package/dist/engine/context.d.ts +81 -0
  10. package/dist/engine/context.js +75 -0
  11. package/dist/engine/copies.d.ts +9 -0
  12. package/dist/engine/copies.js +31 -0
  13. package/dist/engine/planner.d.ts +66 -0
  14. package/dist/engine/planner.js +167 -0
  15. package/dist/engine/repair.d.ts +16 -0
  16. package/dist/engine/repair.js +61 -0
  17. package/dist/engine/sources.d.ts +13 -0
  18. package/dist/engine/sources.js +40 -0
  19. package/dist/engine/translator.d.ts +29 -0
  20. package/dist/engine/translator.js +156 -0
  21. package/dist/engine/writer.d.ts +13 -0
  22. package/dist/engine/writer.js +63 -0
  23. package/dist/files.d.ts +30 -0
  24. package/dist/files.js +125 -12
  25. package/dist/index.d.ts +6 -3
  26. package/dist/index.js +6 -3
  27. package/dist/llm.d.ts +5 -8
  28. package/dist/llm.js +7 -15
  29. package/dist/lock.d.ts +13 -0
  30. package/dist/lock.js +91 -0
  31. package/dist/output.d.ts +21 -0
  32. package/dist/output.js +76 -0
  33. package/dist/placeholders.js +2 -1
  34. package/dist/plugins.d.ts +72 -0
  35. package/dist/plugins.js +71 -0
  36. package/dist/project.d.ts +33 -5
  37. package/dist/project.js +135 -42
  38. package/dist/prompt.d.ts +7 -1
  39. package/dist/prompt.js +10 -3
  40. package/dist/review.d.ts +1 -1
  41. package/dist/review.js +49 -23
  42. package/dist/scope.d.ts +15 -0
  43. package/dist/scope.js +26 -0
  44. package/dist/state.d.ts +11 -4
  45. package/dist/state.js +30 -11
  46. package/dist/translate.d.ts +9 -49
  47. package/dist/translate.js +119 -383
  48. package/dist/ui/data.d.ts +50 -0
  49. package/dist/ui/data.js +178 -0
  50. package/dist/ui/page.d.ts +5 -0
  51. package/dist/ui/page.js +277 -0
  52. package/dist/ui/server.d.ts +24 -0
  53. package/dist/ui/server.js +194 -0
  54. package/dist/util.d.ts +6 -2
  55. package/dist/util.js +17 -5
  56. package/package.json +12 -1
package/README.md CHANGED
@@ -11,7 +11,7 @@ npx localewarden # translate new and changed strings
11
11
  npx localewarden check # quality check, no API calls (use it in CI)
12
12
  ```
13
13
 
14
- Works with i18next, react-intl / FormatJS, vue-i18n, next-intl, ngx-translate and any other setup that keeps strings in JSON files. Uses any OpenAI-compatible API (OpenAI, OpenRouter, a local Ollama, ...).
14
+ Works with i18next, react-intl / FormatJS, vue-i18n, next-intl, ngx-translate and any other setup that keeps strings in JSON files, with Flutter (`.arb` files) and with fastlane's App Store / Play Store metadata (`.txt` files). Uses any OpenAI-compatible API (OpenAI, OpenRouter, a local Ollama, ...).
15
15
 
16
16
  ## Why
17
17
 
@@ -22,19 +22,30 @@ Translating locale files with a language model is easy once. Keeping 20 language
22
22
  - **Models are inconsistent across batches.** French screens mix "tu" and "vous", Spanish copies English Title Case ("Configure Su Cuenta"), and Polish or Russian address every user as a man.
23
23
  - **Broken output ships silently.** A translated placeholder (`{heures}` instead of `{hours}`) shows raw braces in your app. A dropped `</strong>` breaks the layout. Stray Cyrillic letters end up in a Danish sentence.
24
24
 
25
- localewarden grew out of the translation pipeline of a production app that ships in 38 languages. Every rule and check in it exists because one of these failures happened in real output.
25
+ localewarden grew out of the translation pipeline of a production app that ships in 38 languages. Every rule and check in it exists because one of these failures happened in real output. The checks are tuned against that app's real texts (UI, website, long-form learning content and store listings, about 165 MB) to report problems without flooding you with false alarms. On those texts they still find things that slipped through earlier pipelines: sections cut off after the English grew, stray letters from other alphabets, a trial notice left in English.
26
+
27
+ ## How it compares
28
+
29
+ - **Translation platforms** (Crowdin, Lokalise, Phrase, Weblate) are hosted services with editors, translator workflows and review for teams. localewarden is a small CLI that runs in your repository and CI, with no account and no server. If you have professional translators, a platform fits better. If a model translates and people only fix the odd string, this is the lighter setup.
30
+ - **"Translate my JSON with GPT" scripts** usually send every string on every run and overwrite whatever is there. localewarden keeps state, so it only sends what changed, keeps human fixes, and checks the output.
31
+ - **Editor extensions** (such as i18n Ally) help you write and look up keys while coding. localewarden is about filling and maintaining 10 to 40 languages afterwards. The two work well together.
26
32
 
27
33
  ## What it does
28
34
 
29
35
  - **Translates only what changed.** It remembers a hash of each source string per language. New strings are translated. Changed strings are *revised*: the model gets the existing translation and changes only what the source change requires. Removed strings are deleted from every language.
30
36
  - **Protects hand edits.** If someone edited a translation, localewarden detects it, keeps it, and lists it for review. If the source of a hand-edited string changes later, the string is flagged instead of overwritten.
31
- - **Checks every result before writing it.** Broken placeholders, injected HTML or scripts, foreign alphabets, changed links, broken HTML and echoed source text are rejected (retried once, then left for the next run). Softer problems are retried and reported.
37
+ - **Checks every result before writing it.** Broken placeholders, injected HTML or scripts, foreign alphabets, changed links, broken HTML and echoed source text are rejected (retried once, then left for the next run). Softer problems (too long, content missing, words left in English) are retried and reported.
32
38
  - **Consistent style per language.** It enforces formal or informal address per language (`du`/`Sie`, `tu`/`vous`, `ты`/`вы` and 16 more), uses sentence case where the language does, avoids gendered forms for "you", and applies local typography (French spacing, `92 %` in German, CJK quotation marks).
33
39
  - **Plural forms per language.** For i18next-style keys (`item_one`, `item_other`) it adds the forms a language needs but English lacks, such as Polish `_few` and `_many` or Arabic `_zero`, `_two`, `_few` and `_many` (CLDR plural rules).
40
+ - **Data files and store listings.** Fields like `id`, `type` or `image` are copied instead of translated (`ignoreKeys`), and so are URLs, email addresses and file paths. Length limits per key (`maxLength`) are passed to the model and checked: App Store names, SEO titles, buttons.
34
41
  - **Glossary and protected names.** You choose fixed renderings ("Privacy Policy" -> "Politique de confidentialité") and names that must never be translated. The check accepts grammatical case endings.
35
42
  - **Quality check for CI.** `localewarden check` runs all checks without any API calls and exits non-zero on errors.
36
43
  - **Targeted repair.** `--fix-flagged` asks the model to fix only what the check flagged. The fix is accepted only if the problem is gone and little else changed.
37
- - **Budget control.** A token budget per run, a dry run with a cost estimate, and graceful stop and resume.
44
+ - **Budget control.** A token budget per run and per day (for scheduled jobs), a dry run with a cost estimate, and graceful stop and resume.
45
+ - **Groups.** Parts of a project with their own files, languages and model settings, translated in a fixed order (say app UI first, long-form content last, with a cheaper setting).
46
+ - **Plugins.** Your own checks, prompt notes, post-processing and file order, without forking.
47
+ - **Web interface.** `npx localewarden ui`: progress per language, strings with inline editing, check findings, the review list, and runs with a live log. Local only.
48
+ - **Reliable in daily use.** Lock against parallel runs, atomic writes, progress saved on Ctrl+C, Windows line endings and byte order marks kept.
38
49
  - **No runtime dependencies.** Node.js 20+.
39
50
 
40
51
  ## Example
@@ -79,7 +90,7 @@ es Mantén vivas tus plantas sin tener que pensar en ello
79
90
  ja 何も考えなくても、植物を元気に保てます
80
91
  ```
81
92
 
82
- German uses "du" and French "vous", as configured. Spanish and French use sentence case, not the English Title Case. French has its space before "!". The hedge "tend to" survived, and so did the placeholders, the link and the brand name. The full example is in [`examples/basic`](examples/basic).
93
+ German uses "du" and French "vous", as configured. Spanish and French use sentence case, not the English Title Case. French has its space before "!". The hedge "tend to" survived, and so did the placeholders, the link and the brand name. The full example is in [`examples/basic`](examples/basic). There are also examples for [Flutter ARB files](examples/flutter) and [App Store / Play Store texts with fastlane](examples/fastlane).
83
94
 
84
95
  ## Quick start
85
96
 
@@ -111,7 +122,12 @@ German uses "du" and French "vous", as configured. Spanish and French use senten
111
122
  git add locales .localewarden
112
123
  ```
113
124
 
114
- `.localewarden/` holds the hashes that tell localewarden what changed and what was edited by hand. Commit it so your team and your CI share the same state.
125
+ `.localewarden/` holds the hashes that tell localewarden what changed and what was edited by hand. Commit it so your team and your CI share the same state, except two local files:
126
+
127
+ ```gitignore
128
+ .localewarden/usage.json
129
+ .localewarden/run.lock
130
+ ```
115
131
 
116
132
  ## How it decides what to translate
117
133
 
@@ -135,24 +151,28 @@ The same checks run in two places. Right after each model answer, a failed hard
135
151
  | --- | --- | --- |
136
152
  | `placeholder` | `{name}`, `{{count}}`, `%s`, `%1$d`, `%{x}`, `${x}`, `<0></0>` renamed, translated, added or dropped. ICU `plural`/`select` arguments are compared, while plural categories may differ per language. | error |
137
153
  | `unsafe` | HTML tags, attributes, event handlers or `javascript:`/`data:` URLs that the source does not have. Translations are often rendered as raw HTML, so this would be a script injection. | error |
138
- | `script` | Letters from an alphabet the language does not use ("刺激" in German), or a word that mixes Latin with Cyrillic/Greek lookalikes ("Вarda") | error |
154
+ | `script` | Letters from an alphabet the language does not use ("刺激" in German), a word that mixes Latin with Cyrillic/Greek lookalikes ("Вarda"), or Simplified characters in Traditional Chinese (`zh-TW`) and the reverse | error |
139
155
  | `markup` | Changed link targets, different number of tags, unclosed or misnested tags, dropped list items | warning (broken tags and changed links: never written) |
140
156
  | `years` | A year from the source missing or changed (citations, dates) | warning |
141
157
  | `formality` | The other form of address than configured, both forms in one string, or masculine-only forms for "you" | warning |
158
+ | `length` | Longer than the `maxLength` configured for the key | warning |
142
159
  | `titlecase` | English Title Case copied into a language that uses sentence case | warning |
143
160
  | `ampersand` | "&" in languages that write the word | warning |
144
161
  | `glossary` | A glossary rendering missing (case endings allowed), or a `doNotTranslate` name translated | warning |
145
162
  | `untranslated` | Identical to the source (prose of 3+ words; "OK" and names are fine) | warning |
146
- | `partial` | Source-language words left inside the translation, an untranslated bold lead-in, or a hedge that became certainty ("tend to" stated as fact) | warning |
163
+ | `partial` | Source-language words left inside the translation, an untranslated bold lead-in, a hedge that became certainty ("tend to" stated as fact), a translation much shorter than its source (content cut off, or the source grew after it was translated), or sibling options that got the same translation although the source differs ("Rarely" and "Occasionally" both "Selten") | warning |
147
164
 
148
165
  ```bash
149
166
  npx localewarden check # counts per language and check
150
167
  npx localewarden check -v # with examples
151
168
  npx localewarden check --strict # exit 1 on warnings too
152
169
  npx localewarden check --json # for scripts
170
+ npx localewarden check --fix # repair placeholders with one possible fix, no API calls
153
171
  ```
154
172
 
155
- Approved hand edits are skipped, except for errors (placeholder, unsafe, script), which break the app either way.
173
+ `--fix` repairs a translated placeholder when the source has exactly one and the translation renamed it (`{stunden}` back to `{hours}`). The file is edited in place, so its formatting stays as it is. Anything less certain is left for `--fix-flagged` or a person.
174
+
175
+ Approved strings are skipped, except for errors (placeholder, unsafe, script), which break the app either way. You can approve any string, not only hand edits: `npx localewarden review --approve de:home.title` tells the check that a person looked at it (for example a pun on a brand name that is correct without the name), and runs leave it alone.
156
176
 
157
177
  ### In CI
158
178
 
@@ -211,6 +231,22 @@ npx localewarden --fix-flagged
211
231
 
212
232
  For each flagged string, the model gets the source, the current translation and the exact findings, with the instruction to change only what is needed. The fix is written only if the same check passes afterwards, nothing else breaks, and few words changed. Rejected fixes are recorded in `.localewarden/repair-failures.json` and not retried until the translation changes.
213
233
 
234
+ ## Web interface
235
+
236
+ ```bash
237
+ npx localewarden ui # prints a local URL with an access token
238
+ ```
239
+
240
+ - **Overview:** progress per group and language, hand edits waiting for review, tokens used today.
241
+ - **Strings:** search by key, source or translation; show only missing ones; edit a translation inline. An edit counts as checked by a person: it is approved and protected from runs.
242
+ - **Check:** run the quality check and filter findings by language and check.
243
+ - **Review:** approve hand edits or hand them back.
244
+ - **Run:** dry run, translate or fix flagged strings, with a live log.
245
+
246
+ The interface listens on `127.0.0.1` only, needs the random token from the printed URL, and
247
+ rejects requests with another host name, so websites open in your browser cannot use it.
248
+ It writes only to the project's own locale files.
249
+
214
250
  ## Hand edits and review
215
251
 
216
252
  ```bash
@@ -240,6 +276,9 @@ npx localewarden review --release de:home.title # hand it back: next run revis
240
276
  | `glossary` | `{}` | `{"fr": {"Terms of Service": "Conditions d'utilisation"}}` |
241
277
  | `termNotes` | `{}` | Meanings of ambiguous terms, sent only with strings that contain them: `{"snooze": "postpone a reminder"}` |
242
278
  | `instructions` | `{}` | Extra instructions per language, `"*"` for all: `{"es": "Use neutral Latin American Spanish."}` |
279
+ | `ignoreKeys` | `[]` | Keys that are not text, copied from the source: `["id", "type", "**.sources.*"]`. `*` matches within a key segment, `**` across segments; a pattern without a dot matches the last segment anywhere. URLs, emails, file paths and numbers are always copied. |
280
+ | `exclude` | `[]` | Source files to skip: `["locales/{lang}/nav.json"]` |
281
+ | `maxLength` | `{}` | Character limits per key pattern: `{"**.meta.title": 60, "name": 30}`. The model is told the limit; longer results are retried once and reported by the `length` check. |
243
282
  | `placeholders` | built-in | Regular expressions (strings) that match your placeholders. Replaces the built-in list. |
244
283
  | `model` | `"gpt-5.4-mini"` | Any chat model your endpoint offers |
245
284
  | `baseUrl` | `"https://api.openai.com/v1"` | Any OpenAI-compatible endpoint |
@@ -250,6 +289,102 @@ npx localewarden review --release de:home.title # hand it back: next run revis
250
289
  | `concurrency` | `4` | Languages translated in parallel |
251
290
  | `batchSize` | `20` | Strings per request (smaller for scripts that need many tokens) |
252
291
  | `stateDir` | `".localewarden"` | Where state and the review list live |
292
+ | `dailyTokenBudget` | | Token limit per UTC day across runs (for scheduled jobs); usage is kept in `<stateDir>/usage.json` (add it to `.gitignore`) |
293
+ | `copies` | `{}` | Locales that are a copy of another one instead of a translation: `{"en-GB": "en-US", "fr-CA": "fr-FR"}` |
294
+ | `chunkChars` | `8000` | Longer strings are translated paragraph by paragraph |
295
+ | `plugins` | `[]` | Plugin modules, see [Plugins](#plugins) |
296
+ | `groups` | | Parts of the project with their own settings, see [Groups](#groups) |
297
+
298
+ ### Groups
299
+
300
+ Different parts of a project often need different settings: the app UI translated first and
301
+ with care, long articles last and with a cheaper setting, store listings with other locale
302
+ codes. Each group inherits the top-level settings and may override them:
303
+
304
+ ```json
305
+ {
306
+ "targetLanguages": ["de", "fr", "pl"],
307
+ "dailyTokenBudget": 2000000,
308
+ "groups": [
309
+ { "name": "app", "files": "src/locales/{lang}/*.json" },
310
+ { "name": "store", "files": "fastlane/metadata/{lang}/*.txt", "sourceLanguage": "en-US",
311
+ "targetLanguages": ["de-DE", "fr-FR", "pl"], "copies": { "fr-CA": "fr-FR" } },
312
+ { "name": "articles", "files": "content/*/*.{lang}.json", "reasoningEffort": "low",
313
+ "ignoreKeys": ["id", "slug", "image"] }
314
+ ]
315
+ }
316
+ ```
317
+
318
+ Groups run in this order and share the budget, so the important ones are done first when the
319
+ budget runs out. `--group app,store` runs only some; an unknown group name is an error.
320
+
321
+ What a group inherits: maps (`formality`, `glossary` per language, `termNotes`, `instructions`,
322
+ `maxLength`) are merged with the top level, lists (`doNotTranslate`, `ignoreKeys`, `exclude`)
323
+ are extended, and everything else is replaced. A group with its own `targetLanguages` or
324
+ `sourceLanguage` does not inherit `copies`, since those name locales. `stateDir`, `plugins`, `dailyTokenBudget`,
325
+ `maxTokensPerRun` and `concurrency` apply to the whole run and can only be set at the top level.
326
+
327
+ ### Plugins
328
+
329
+ A plugin adds project rules without forking localewarden. It is an ES module; its default
330
+ export is a plugin object, or a function that receives the options from the config:
331
+
332
+ ```js
333
+ // rules/my-plugin.mjs
334
+ export default (options) => ({
335
+ name: 'my-rules',
336
+ // Extra findings for one translated string. "error" blocks writing it and fails `check`;
337
+ // "fixable" lets --fix-flagged repair it.
338
+ checks: ({ lang, key, file, source, text }) =>
339
+ lang === 'es' && /\bcoger\b/i.test(text)
340
+ ? [{ check: 'regional-term', note: 'use "tomar"', fixable: true }]
341
+ : [],
342
+ // Extra prompt text for a batch (sent with every request of the batch).
343
+ promptNotes: ({ lang, items }) => (lang === 'es' ? 'Use neutral Latin American Spanish.' : ''),
344
+ // Rewrites a model answer before it is checked and written.
345
+ postProcess: ({ lang, text }) => (lang === 'fr' ? text.replace(/ ([?!:;])/g, '\u00a0$1') : text),
346
+ // Reorders or filters the files of a group.
347
+ order: (files, { group }) => files,
348
+ });
349
+ ```
350
+
351
+ ```json
352
+ { "plugins": ["./rules/my-plugin.mjs", { "module": "./rules/blog.mjs", "options": { "draftsFolder": "drafts" } }] }
353
+ ```
354
+
355
+ All hooks are optional. The interface is marked experimental in 0.x and may change in a minor
356
+ version. A complete example is in [`examples/plugin`](examples/plugin).
357
+
358
+ ### App Store and Play Store listings (fastlane)
359
+
360
+ ```json
361
+ {
362
+ "sourceLanguage": "en-US",
363
+ "targetLanguages": ["de-DE", "fr-FR", "ja"],
364
+ "files": "fastlane/metadata/{lang}/*.txt",
365
+ "exclude": ["fastlane/metadata/{lang}/*_url.txt"],
366
+ "maxLength": { "name": 30, "subtitle": 30, "keywords": 100, "promotional_text": 170, "description": 4000 },
367
+ "termNotes": { "keywords": "a comma-separated keyword list for store search, not a sentence" }
368
+ }
369
+ ```
370
+
371
+ Each `.txt` file is one string, keyed by its file name, so the limits above apply to `name.txt`, `subtitle.txt` and so on.
372
+
373
+ ### Flutter (ARB)
374
+
375
+ ```json
376
+ { "files": "lib/l10n/app_{lang}.arb", "targetLanguages": ["de", "fr", "pt_BR"] }
377
+ ```
378
+
379
+ Metadata (`@@locale`, `@key` descriptions and placeholders) is copied, not translated, and `@@locale` is set to the target language. ICU plurals and selects keep their structure, and each language gets the plural categories it needs.
380
+
381
+ ### Data files
382
+
383
+ For content JSON with ids, types and links, list the non-text keys:
384
+
385
+ ```json
386
+ { "files": "content/**/*.{lang}.json", "ignoreKeys": ["id", "type", "category", "image", "**.sources.*"] }
387
+ ```
253
388
 
254
389
  ### Other providers
255
390
 
@@ -266,10 +401,12 @@ Small local models make noticeably more mistakes. The checks catch the mechanica
266
401
  ## Commands
267
402
 
268
403
  ```text
269
- localewarden [translate] --dry-run --lang de,fr --fix-flagged --retranslate-all
404
+ localewarden [translate] --dry-run --lang de,fr --group app --fix-flagged --retranslate-all
405
+ --retranslate-files <patterns> --refresh-before <YYYY-MM-DD>
270
406
  --overwrite-manual --max-tokens <n> --verbose
271
- localewarden check --lang de,fr --verbose --limit <n> --strict --json
407
+ localewarden check --lang de,fr --group app --verbose --limit <n> --strict --json --fix
272
408
  localewarden review --all --approve <sel>... --release <sel>...
409
+ localewarden ui --port <n>
273
410
  localewarden init
274
411
  Global: --config <path> --help --version
275
412
  ```
@@ -300,11 +437,23 @@ Translations are treated as untrusted: any markup the source does not have is bl
300
437
 
301
438
  ## Limitations
302
439
 
303
- - JSON only (nested objects, arrays, flat keys). YAML, PO, XLIFF and ARB are not supported yet.
440
+ - JSON (nested objects, arrays, flat keys), Flutter ARB and plain `.txt` files. YAML, PO and XLIFF are not supported yet.
304
441
  - The checks catch mechanical problems, not every wrong meaning. Have a native speaker look at important screens, then approve their edits with `review`.
305
442
  - Rules for form of address, gender and typography exist for the languages listed above. Other languages are translated with the general rules.
306
443
  - A run that is interrupted keeps everything written so far. Unwritten strings are picked up on the next run.
307
444
 
445
+ ## Reliability
446
+
447
+ - **One run at a time:** a lock file in the state folder stops a second run (say CI and a local run), or an approval or edit during a run, from writing the same state; a lock left by a crashed process (or committed from another machine) is taken over safely.
448
+ - **Errors stop cleanly:** when one language fails, no further language starts, and the run ends only after the running ones finished.
449
+ - **Symbolic links** to locale files are written through; the file keeps its permissions.
450
+ - **No half-written files:** locale files and state are written to a temporary file and renamed.
451
+ - **Ctrl+C or a CI timeout** saves what was translated so far.
452
+ - **File formats are kept:** indentation, key order of existing files, Windows line endings, byte order marks. A file whose content did not change is not rewritten.
453
+ - **Unicode:** text is compared NFC-normalized, so an editor that saves "é" decomposed does not look like a hand edit.
454
+ - **Clear errors** for invalid JSON, a corrupt state file, an unknown group, an old Node.js version, or a dotted key that collides with a nested one.
455
+ - **Symbolic links** to locale folders are followed (once, so a link loop does no harm).
456
+
308
457
  ## Contributing
309
458
 
310
459
  Bug reports with a concrete example (source, language, output, expected) help most. See [CONTRIBUTING.md](CONTRIBUTING.md).
@@ -0,0 +1,25 @@
1
+ export declare class BudgetExceededError extends Error {
2
+ }
3
+ /**
4
+ * Token budget shared by every request of a run: a limit for the run and, optionally, a
5
+ * limit per UTC day that holds across runs (for scheduled jobs). Daily usage is kept in a
6
+ * small JSON file; a run that crosses midnight starts counting the new day.
7
+ */
8
+ export declare class Budget {
9
+ readonly maxRunTokens: number;
10
+ readonly dailyTokens?: number | undefined;
11
+ private readonly usageFile?;
12
+ /** Tokens and requests of this run. */
13
+ tokens: number;
14
+ requests: number;
15
+ private usage;
16
+ constructor(maxRunTokens: number, dailyTokens?: number | undefined, usageFile?: string | undefined);
17
+ private load;
18
+ /** Tokens used today, including earlier runs. */
19
+ get usedToday(): number;
20
+ /** Why no further request may start, or null. */
21
+ get exceeded(): string | null;
22
+ /** Throws when the budget is used up. Called before each request. */
23
+ check(): void;
24
+ record(tokens: number): void;
25
+ }
package/dist/budget.js ADDED
@@ -0,0 +1,74 @@
1
+ import fs from 'node:fs';
2
+ import { writeText } from './files.js';
3
+ export class BudgetExceededError extends Error {
4
+ }
5
+ const utcDay = () => new Date().toISOString().slice(0, 10);
6
+ /**
7
+ * Token budget shared by every request of a run: a limit for the run and, optionally, a
8
+ * limit per UTC day that holds across runs (for scheduled jobs). Daily usage is kept in a
9
+ * small JSON file; a run that crosses midnight starts counting the new day.
10
+ */
11
+ export class Budget {
12
+ maxRunTokens;
13
+ dailyTokens;
14
+ usageFile;
15
+ /** Tokens and requests of this run. */
16
+ tokens = 0;
17
+ requests = 0;
18
+ usage;
19
+ constructor(maxRunTokens, dailyTokens, usageFile) {
20
+ this.maxRunTokens = maxRunTokens;
21
+ this.dailyTokens = dailyTokens;
22
+ this.usageFile = usageFile;
23
+ this.usage = this.load();
24
+ }
25
+ load() {
26
+ if (this.usageFile) {
27
+ try {
28
+ const data = JSON.parse(fs.readFileSync(this.usageFile, 'utf8'));
29
+ if (data.date === utcDay())
30
+ return data;
31
+ }
32
+ catch {
33
+ // missing or unreadable: start the day at zero
34
+ }
35
+ }
36
+ return { date: utcDay(), tokens: 0, requests: 0 };
37
+ }
38
+ /** Tokens used today, including earlier runs. */
39
+ get usedToday() {
40
+ if (this.usage.date !== utcDay())
41
+ this.usage = { date: utcDay(), tokens: 0, requests: 0 };
42
+ return this.usage.tokens;
43
+ }
44
+ /** Why no further request may start, or null. */
45
+ get exceeded() {
46
+ if (this.tokens >= this.maxRunTokens)
47
+ return `token budget of ${this.maxRunTokens.toLocaleString('en')} for this run reached`;
48
+ if (this.dailyTokens !== undefined && this.usedToday >= this.dailyTokens) {
49
+ return `daily token budget of ${this.dailyTokens.toLocaleString('en')} reached (${this.usedToday.toLocaleString('en')} used today)`;
50
+ }
51
+ return null;
52
+ }
53
+ /** Throws when the budget is used up. Called before each request. */
54
+ check() {
55
+ const reason = this.exceeded;
56
+ if (reason)
57
+ throw new BudgetExceededError(reason);
58
+ }
59
+ record(tokens) {
60
+ this.tokens += tokens;
61
+ this.requests++;
62
+ this.usedToday; // rolls the day over if midnight passed
63
+ this.usage.tokens += tokens;
64
+ this.usage.requests++;
65
+ if (this.usageFile) {
66
+ try {
67
+ writeText(this.usageFile, JSON.stringify(this.usage, null, 2) + '\n');
68
+ }
69
+ catch {
70
+ // not fatal: the run limit still applies
71
+ }
72
+ }
73
+ }
74
+ }
package/dist/checks.d.ts CHANGED
@@ -1,4 +1,6 @@
1
1
  import type { Config } from './config.js';
2
+ import { PluginHost } from './plugins.js';
3
+ import { Scope } from './scope.js';
2
4
  /**
3
5
  * Deterministic quality checks. No API calls, so they can run in CI on every commit.
4
6
  *
@@ -11,31 +13,50 @@ import type { Config } from './config.js';
11
13
  * years a year from the source is missing or changed (citations, dates)
12
14
  * formality the other form of address than configured, or both mixed;
13
15
  * masculine-only forms for "you" when genderNeutral is on
16
+ * length longer than the maxLength configured for the key
14
17
  * titlecase English Title Case copied into a sentence-case language
15
18
  * ampersand "&" in a language that writes the word
16
19
  * glossary a glossary rendering or a doNotTranslate name is missing
17
20
  * untranslated identical to the source (prose of 3+ words)
18
21
  * partial source-language words left inside an otherwise translated string,
19
- * or a dropped hedge ("tend to" stated as certain)
22
+ * a dropped hedge ("tend to" stated as certain), or a translation much
23
+ * shorter than its source (content missing or cut off)
20
24
  */
21
- export type CheckName = 'placeholder' | 'unsafe' | 'script' | 'markup' | 'years' | 'formality' | 'titlecase' | 'ampersand' | 'glossary' | 'untranslated' | 'partial';
25
+ export type CheckName = 'placeholder' | 'unsafe' | 'script' | 'markup' | 'years' | 'formality' | 'length' | 'titlecase' | 'ampersand' | 'glossary' | 'untranslated' | 'partial';
22
26
  export declare const CHECKS: CheckName[];
23
27
  export declare const ERROR_CHECKS: Set<CheckName>;
24
28
  /** Checks a targeted repair (--fix-flagged) may try to fix. */
25
29
  export declare const FIXABLE_CHECKS: Set<CheckName>;
26
30
  export interface Issue {
27
- check: CheckName;
31
+ /** A built-in check name, or a plugin's own check name. */
32
+ check: CheckName | (string & {});
28
33
  note?: string;
34
+ /** Set for plugin issues; built-in checks use ERROR_CHECKS. */
35
+ severity?: 'error' | 'warning';
36
+ /** Set for plugin issues; built-in checks use FIXABLE_CHECKS. */
37
+ fixable?: boolean;
29
38
  }
39
+ /** Errors break the software (and fail `localewarden check`); everything else is a warning. */
40
+ export declare const isError: (issue: Issue) => boolean;
41
+ /** Whether `--fix-flagged` may ask the model to fix the issue. */
42
+ export declare const isFixable: (issue: Issue) => boolean;
30
43
  export declare class Checker {
31
44
  readonly config: Config;
32
45
  readonly placeholderRe: RegExp;
33
- constructor(config: Config);
46
+ readonly scope: Scope;
47
+ /** Words of termNotes and doNotTranslate: terms a translation may keep in the source language. */
48
+ readonly keptWords: Set<string>;
49
+ readonly plugins: PluginHost;
50
+ constructor(config: Config, plugins?: PluginHost);
51
+ /** The text without doNotTranslate names, which stay the same in every language. */
52
+ withoutNames(text: string): string;
53
+ /** "62 characters, limit 60", or null. */
54
+ tooLong(key: string, text: string): string | null;
34
55
  get englishSource(): boolean;
35
56
  placeholdersMatch(key: string, source: string, text: string): boolean;
36
57
  placeholderNote(source: string, text: string): string;
37
58
  /** All issues of one translated string. */
38
- checkString(lang: string, key: string, source: string, text: string): Issue[];
59
+ checkString(lang: string, key: string, source: string, text: string, file?: string): Issue[];
39
60
  /**
40
61
  * Output check right after the API call.
41
62
  * hard: certain corruption (placeholders, foreign alphabet, changed link targets, broken
@@ -43,13 +64,15 @@ export declare class Checker {
43
64
  * soft: likely loss (tag count, missing year, source words left in). Retried once, then
44
65
  * accepted; `localewarden check` keeps reporting it.
45
66
  */
46
- defect(lang: string, key: string, source: string, text: string): {
67
+ defect(lang: string, key: string, source: string, text: string, file?: string): {
47
68
  hard: string | null;
48
69
  soft: string | null;
49
70
  };
50
71
  }
51
72
  /** Non-Latin scripts each language is written in. Latin is always allowed (names, codes). */
52
73
  export declare const NATIVE_SCRIPTS: Record<string, string[]>;
74
+ /** Simplified characters in Traditional Chinese text, or the reverse (2+ distinct ones). */
75
+ export declare function wrongChineseScript(lang: string, text: string): string | null;
53
76
  /** Why `text` contains letters that cannot belong to `lang`, or null. */
54
77
  export declare function foreignScript(lang: string, text: string): string | null;
55
78
  /**
@@ -82,3 +105,8 @@ export declare function isUnchangedProse(source: string, text: string, placehold
82
105
  * allowed) copied into the translation, or null.
83
106
  */
84
107
  export declare function sourceRun(source: string, text: string): string | null;
108
+ /**
109
+ * A translation with a fraction of the source's length lost content: cut off by the model, or
110
+ * the source grew after it was translated. Markup and placeholders are not counted.
111
+ */
112
+ export declare function muchShorter(lang: string, source: string, text: string): string | null;