locallingo 0.4.3 → 0.5.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +74 -3
- data/README.md +94 -0
- data/config/locallingo.default.yml +11 -1
- data/lib/locallingo/active_record.rb +148 -0
- data/lib/locallingo/batch_translator.rb +151 -0
- data/lib/locallingo/cli.rb +44 -10
- data/lib/locallingo/configuration.rb +4 -1
- data/lib/locallingo/content_leaves.rb +40 -0
- data/lib/locallingo/drift.rb +93 -0
- data/lib/locallingo/json_extraction.rb +14 -2
- data/lib/locallingo/manager.rb +68 -121
- data/lib/locallingo/prompt.rb +73 -0
- data/lib/locallingo/providers/ruby_llm.rb +36 -9
- data/lib/locallingo/quality_checker.rb +31 -6
- data/lib/locallingo/record_translator.rb +91 -0
- data/lib/locallingo/translator.rb +70 -0
- data/lib/locallingo/validators/duplicate_values.rb +4 -4
- data/lib/locallingo/validators/manual_edits.rb +4 -8
- data/lib/locallingo/validators/outdated.rb +5 -5
- data/lib/locallingo/version.rb +1 -1
- data/lib/locallingo.rb +5 -0
- metadata +10 -3
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: b218e7422188db797e061b9cf1133b1b25a630e9e4c6237c04f0f7fa94dd2b4f
|
|
4
|
+
data.tar.gz: 742d0794ade8f32f9346d4a02b61554b7cb81321569c5297f9d93c3f1639f4e3
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 48e0a2e2625a03fc0d78e9b5a9884efc4ec0ee7525533f150865d7bb391c473edd96f44bbca6974a24a6e564d4d08e644a0b36403a5c8a51b28a0943d54c1d1c
|
|
7
|
+
data.tar.gz: 4f08c9b89ab57351c5099ff6e31c4c24fa6b33a16df8ec0b70cb0ba365e292d90ef0fbe047d4b5494b9c2c7abe0627657c6ce4b545fd4a4552150f7338fd159d
|
data/CHANGELOG.md
CHANGED
|
@@ -6,12 +6,81 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
|
6
6
|
|
|
7
7
|
## [Unreleased]
|
|
8
8
|
|
|
9
|
+
### Added
|
|
10
|
+
- `Locallingo::Translator#translate(texts, from:, to:, style:, instructions:)`
|
|
11
|
+
translates a plain `{ key => text }` hash with the configured prompt,
|
|
12
|
+
glossary, language guide, provider and model, and returns `translations` and
|
|
13
|
+
`failed`. (#33)
|
|
14
|
+
- `Locallingo::Prompt.translation`: the translation prompt, public. `style:
|
|
15
|
+
:prose` keeps Markdown structure and drops the UI brevity rule;
|
|
16
|
+
`instructions:` is appended after the language guide. The CLI's `:ui`
|
|
17
|
+
prompt is unchanged. (#33)
|
|
18
|
+
- `Providers::RubyLLM.new(provider:, context:)` and `Translator.new(context:)`
|
|
19
|
+
take a `RubyLLM::Context`; with one, the global `RubyLLM.config` is never
|
|
20
|
+
written. (#33)
|
|
21
|
+
- `translate.max_batch_chars` limits the source characters per batch (a single
|
|
22
|
+
value over the limit is sent alone, never split); `translate.pause` sets the
|
|
23
|
+
sleep after each batch (default `1.0`). (#33)
|
|
24
|
+
- `Locallingo::Drift`: missing / outdated / manual / hand-edited decisions as a
|
|
25
|
+
value over the state entry shape, for strings and nested values, with a
|
|
26
|
+
pluggable hasher. `Manager` and the validators use it; behaviour is
|
|
27
|
+
unchanged. (#33)
|
|
28
|
+
- `require "locallingo/active_record"` adds `Locallingo::Translatable`
|
|
29
|
+
(`translates_content`), and `Locallingo::RecordTranslator` translates many
|
|
30
|
+
records in one batched call, writing each under its row lock. (#33)
|
|
31
|
+
- `lingo quality --prefix yoga.` checks only the keys under a prefix, in both
|
|
32
|
+
the static checks and the AI pass. A prefix that matches nothing exits `1`. (#10, #29)
|
|
33
|
+
- `lingo quality --key a.b` (repeatable) checks exactly the named keys, and the
|
|
34
|
+
AI pass reviews every named key without sampling. An unknown key exits `1`;
|
|
35
|
+
`--key` and `--prefix` cannot be combined. (#10, #29)
|
|
36
|
+
- `quality.sample_size` (default `100`) sets how many keys `quality --ai`
|
|
37
|
+
samples. (#10, #29)
|
|
38
|
+
- `lingo translate --include-manual` lets `--force-key` overwrite keys flagged
|
|
39
|
+
`manual`; each overwritten key is named on stderr and its `manual` flag
|
|
40
|
+
survives. (#11, #30)
|
|
41
|
+
|
|
9
42
|
### Changed
|
|
10
|
-
-
|
|
11
|
-
|
|
12
|
-
|
|
43
|
+
- `lingo translate --force-key` skips keys flagged `manual` (hand-edited, via
|
|
44
|
+
`accept-edits`) and names each skipped key on stderr, the same protection
|
|
45
|
+
`--force` already gave. If every named key is manual, nothing is translated.
|
|
46
|
+
Pass `--include-manual` to overwrite them as before. (#11, #30)
|
|
47
|
+
- The translation prompt asks for natural, idiomatic text in the target
|
|
48
|
+
language ("into natural, idiomatic German") instead of a word-for-word
|
|
49
|
+
translation: the meaning, phrasing and UI conventions a native speaker would
|
|
50
|
+
use, with idioms replaced by their local equivalent. Only newly translated or
|
|
51
|
+
outdated keys are affected; run `lingo translate --force` to redo all
|
|
52
|
+
non-manual keys (hand-edited `manual` keys stay untouched — redo those one at
|
|
53
|
+
a time with `--force-key <key> --include-manual` and compare).
|
|
54
|
+
The source language is now named too (`en` → "English").
|
|
55
|
+
- The translation prompt no longer hardcodes "Use formal business language".
|
|
56
|
+
Register (formal "Sie"/"Lei" vs. informal "du"/"tu") belongs in each
|
|
57
|
+
locale's `language_guides` entry; add it there to keep a formal tone.
|
|
58
|
+
- `lingo translate` exits `1` when any key fails to translate, and prints
|
|
59
|
+
`⚠️ Translation finished: N keys failed` with the failed keys instead of
|
|
60
|
+
`✅ Translation complete!`. Translated keys are still written and the
|
|
61
|
+
`after_translate` hooks still run; `--dry-run` reports failures but exits `0`.
|
|
62
|
+
Appending `|| true` restores the old always-`0` behaviour, but also hides
|
|
63
|
+
every other failure (missing credentials, a broken config). (#25)
|
|
64
|
+
- `Manager#translate!` returns `{ locale => failed_keys }`. (#25)
|
|
65
|
+
- Require `ruby_llm` >= 2.1 (constraint is now `>= 2.1, < 3`; host apps pinned
|
|
66
|
+
to RubyLLM 1.x or 2.0 must bump). The translation and quality-review system
|
|
67
|
+
prompts are marked as prompt cache boundaries; the translation prompt is
|
|
68
|
+
reused by every batch of a locale, so providers with prompt caching
|
|
69
|
+
(Anthropic, OpenAI-compatible) bill that repeated prefix at the cached rate. Anthropic only caches prompts
|
|
70
|
+
above its minimum length (~1024 tokens); shorter prompts are sent uncached.
|
|
13
71
|
|
|
14
72
|
### Fixed
|
|
73
|
+
- `lingo quality --ai` includes the locale's `language_guides` entry in the
|
|
74
|
+
review prompt, as `translate` does, so the AI pass can flag violations of the
|
|
75
|
+
guide (e.g. du/Sie register). (#9, #29)
|
|
76
|
+
- One invalid value in a model reply (e.g. typographic quotes returned as
|
|
77
|
+
unescaped `"`) no longer fails every key in its batch. A reply that cannot be
|
|
78
|
+
parsed is never resent as-is: the batch is split in half, down to single keys,
|
|
79
|
+
so only the bad key fails. Transport errors still retry with backoff. (#25)
|
|
80
|
+
- When a ```json fence holds invalid JSON, the parse error now names the line
|
|
81
|
+
and column inside the fence instead of the fence at line 1. (#25)
|
|
82
|
+
- The translation prompt asks the model to escape `"` inside values as `\"`
|
|
83
|
+
and keep typographic quotes (“ ” „ ‚ « ») as they appear in the source. (#25)
|
|
15
84
|
- State files are written with a trailing final newline, so the output is
|
|
16
85
|
POSIX-conformant and files that already end with a newline are no longer
|
|
17
86
|
rewritten with newline-only diffs. One-time effect after upgrading: each
|
|
@@ -21,6 +90,8 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
|
21
90
|
through `translate` or `accept-edits`) get a baseline and the `manual_edits`
|
|
22
91
|
validator can watch them. An existing `target_hash` is still never
|
|
23
92
|
recomputed — that would silently absorb hand-edit drift.
|
|
93
|
+
- The `duplicate_values` validator reports findings under the configured
|
|
94
|
+
`source_locale` instead of a hard-coded `en`. (#15)
|
|
24
95
|
|
|
25
96
|
### Fixed (0.4.0)
|
|
26
97
|
- `lingo sync` no longer destroys hand-edit protection: it now only refreshes
|
data/README.md
CHANGED
|
@@ -100,9 +100,11 @@ With no `packages:`, the whole app uses `defaults` — the common case.
|
|
|
100
100
|
lingo status # translation status per locale
|
|
101
101
|
lingo translate --locale de # translate missing/changed keys
|
|
102
102
|
lingo translate --force # re-translate everything
|
|
103
|
+
lingo translate --force-key a.b.c # re-translate one key (skips manual; add --include-manual)
|
|
103
104
|
lingo validate --strict # CI gate (exit 1 on strict-tier issues)
|
|
104
105
|
lingo validate --strict-all # stricter CI gate
|
|
105
106
|
lingo quality --ai # quality linting (+ optional AI pass)
|
|
107
|
+
lingo quality --ai --prefix yoga. # lint one namespace
|
|
106
108
|
lingo fix-quality --locale en # auto-fix fixable issues
|
|
107
109
|
lingo accept-edits --locale de # protect hand-edited translations (drifted keys only)
|
|
108
110
|
lingo accept-edits --key a.b.c # protect a specific key
|
|
@@ -144,6 +146,98 @@ Locallingo::Manager.new(config:).validate # => [violations]
|
|
|
144
146
|
Locallingo::QualityChecker.new(config:).check(locale: "en")
|
|
145
147
|
```
|
|
146
148
|
|
|
149
|
+
### Translating text that is not in locale files
|
|
150
|
+
|
|
151
|
+
`Locallingo::Translator` runs the same prompt (rules, glossary, language guide),
|
|
152
|
+
provider, model and batching over a plain `{ key => text }` hash:
|
|
153
|
+
|
|
154
|
+
```ruby
|
|
155
|
+
require "ruby_llm" # for RubyLLM.context (Bundler.require already loads it in Rails)
|
|
156
|
+
|
|
157
|
+
translator = Locallingo::Translator.new(
|
|
158
|
+
config: Locallingo.configuration(root_path: Dir.pwd),
|
|
159
|
+
context: RubyLLM.context { |c| c.anthropic_api_key = tenant_key }, # optional, per-tenant key
|
|
160
|
+
pause: 0 # no sleep between batches
|
|
161
|
+
)
|
|
162
|
+
|
|
163
|
+
result = translator.translate(
|
|
164
|
+
{ "title" => "Hello", "intro" => "Some **markdown**." },
|
|
165
|
+
from: "en", to: "de",
|
|
166
|
+
style: :prose, # :ui (default) or :prose (keeps Markdown, no brevity rule)
|
|
167
|
+
instructions: "Write in a warm, plain voice." # appended after the language guide
|
|
168
|
+
)
|
|
169
|
+
result.translations # => { "title" => "Hallo", "intro" => "Etwas **Markdown**." }
|
|
170
|
+
result.failed # => keys that failed after retries
|
|
171
|
+
```
|
|
172
|
+
|
|
173
|
+
- With a `context:`, every call uses that `RubyLLM::Context`'s config; the
|
|
174
|
+
global `RubyLLM.config` is never written, so tenants can translate
|
|
175
|
+
concurrently.
|
|
176
|
+
- `MissingCredentialsError` is raised before any call when no key is found.
|
|
177
|
+
- `translate.max_batch_chars` (or `max_batch_chars:`) limits the source
|
|
178
|
+
characters per batch (a single longer value is sent alone, not split); `translate.pause` (or `pause:`) is the sleep after each
|
|
179
|
+
batch. Retries of failed or omitted keys keep their own backoff.
|
|
180
|
+
- `Locallingo::Prompt.translation(config:, from:, to:, style:, instructions:)`
|
|
181
|
+
returns the prompt itself.
|
|
182
|
+
|
|
183
|
+
`Locallingo::Drift` is the missing / outdated / hand-edited rule over the state
|
|
184
|
+
entry shape (`source_hash`, `target_hash`, `manual`), for strings or nested
|
|
185
|
+
values:
|
|
186
|
+
|
|
187
|
+
```ruby
|
|
188
|
+
drift = Locallingo::Drift.new # hasher: any callable(String) -> String
|
|
189
|
+
drift.status(source:, target:, entry:) # => :missing | :outdated | :current
|
|
190
|
+
drift.due?(source:, target:, entry:, force: false, include_manual: false)
|
|
191
|
+
drift.hand_edited?(target:, entry:)
|
|
192
|
+
drift.record(source:, target:, manual: false) # => a new entry
|
|
193
|
+
```
|
|
194
|
+
|
|
195
|
+
### Translating records
|
|
196
|
+
|
|
197
|
+
`require "locallingo/active_record"` (ActiveRecord >= 7.1; nothing loads
|
|
198
|
+
ActiveRecord otherwise) adds a concern for a JSON(B) column keyed by locale
|
|
199
|
+
plus a JSON(B) state column:
|
|
200
|
+
|
|
201
|
+
```ruby
|
|
202
|
+
class Article < ApplicationRecord
|
|
203
|
+
include Locallingo::Translatable
|
|
204
|
+
|
|
205
|
+
translates_content :body, # { "en" => { … }, "de" => { … } }
|
|
206
|
+
state: :translation_state, # { "de" => { "source_hash", "target_hash", "manual" } }
|
|
207
|
+
source_locale: -> { "en" }, # String/Symbol, or a callable run on the record
|
|
208
|
+
target_locales: -> { %w[de sv] },
|
|
209
|
+
translatable: ->(path, value) { true } # optional filter on flattened String leaves
|
|
210
|
+
end
|
|
211
|
+
|
|
212
|
+
article.translation_status(:de) # => :missing | :outdated | :current
|
|
213
|
+
article.translation_manual?(:de)
|
|
214
|
+
article.translation_due?(:de, force: false)
|
|
215
|
+
article.record_translation(:de, manual: false)
|
|
216
|
+
|
|
217
|
+
Locallingo::RecordTranslator.new(translator:)
|
|
218
|
+
.call(Article.where(id: ids), to: "de", style: :prose)
|
|
219
|
+
# => { translated: 3, failed: { article_id => [paths] }, skipped: [ids] }
|
|
220
|
+
```
|
|
221
|
+
|
|
222
|
+
- Saving a record runs `mark_hand_edits`: a target locale a person changed is
|
|
223
|
+
flagged `manual` and never machine-overwritten (`include_manual: true`
|
|
224
|
+
overrides, keeping the flag). That includes a target locale saved for the
|
|
225
|
+
first time, e.g. a source copy used as a placeholder at `create!`: leave
|
|
226
|
+
untranslated locales out of the column instead. A cleared locale is not
|
|
227
|
+
flagged and is due again, unless it was already `manual`: the flag stays, so
|
|
228
|
+
retranslating it needs `include_manual: true`.
|
|
229
|
+
- `RecordTranslator` sends one batched translation per source locale across all
|
|
230
|
+
due records, then writes each record under `record.with_lock`. A record whose
|
|
231
|
+
source changed, or whose target became manual, while the model answered is
|
|
232
|
+
skipped (`skipped:`) and left for the next run, as is a due record with no
|
|
233
|
+
translatable text.
|
|
234
|
+
- Nested values keep their shape; leaves the `translatable:` filter rejects are
|
|
235
|
+
copied from the source. A record with any segment that failed after retries
|
|
236
|
+
is not written at all: its paths are reported in `failed:`, and it stays due
|
|
237
|
+
for the next run (your fallback to the source locale applies meanwhile).
|
|
238
|
+
- Writes use `update_columns`; pass `write: ->(record, attributes) { … }` to
|
|
239
|
+
bump a lock version or run validations.
|
|
240
|
+
|
|
147
241
|
## License
|
|
148
242
|
|
|
149
243
|
MIT © Mikael Henriksson
|
|
@@ -26,9 +26,18 @@ defaults:
|
|
|
26
26
|
translate:
|
|
27
27
|
model: gpt-5-mini
|
|
28
28
|
batch_size: 20
|
|
29
|
+
# Optional cap on the characters of source text per batch (a batch closes at
|
|
30
|
+
# batch_size keys or max_batch_chars, whichever comes first). Useful for long
|
|
31
|
+
# prose; a single oversized value is still sent alone. null = keys only.
|
|
32
|
+
max_batch_chars: null
|
|
33
|
+
# Seconds to sleep after each batch (rate-limit courtesy). 0 disables it.
|
|
34
|
+
pause: 1.0
|
|
29
35
|
|
|
30
36
|
quality:
|
|
31
37
|
model: gpt-4o-mini
|
|
38
|
+
# How many randomly sampled keys `quality --ai` sends for review per run
|
|
39
|
+
# (`--key` reviews every named key instead).
|
|
40
|
+
sample_size: 100
|
|
32
41
|
# Flag en-only British-vs-American spelling drift (organization -> organisation).
|
|
33
42
|
british_spellings: false
|
|
34
43
|
# Named built-in terminology list ("business" | "banking" | "none"), or a
|
|
@@ -41,7 +50,8 @@ defaults:
|
|
|
41
50
|
context: "a business application"
|
|
42
51
|
placeholder_style: "%<name>s, %<count>s"
|
|
43
52
|
glossary: {}
|
|
44
|
-
# Extra per-locale style guidance appended to the translation
|
|
53
|
+
# Extra per-locale style guidance appended to the translation and AI quality
|
|
54
|
+
# review prompts. Value is
|
|
45
55
|
# inline text or `file: path/to/guide.md`.
|
|
46
56
|
language_guides: {}
|
|
47
57
|
|
|
@@ -0,0 +1,148 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "active_support/concern"
|
|
4
|
+
require "active_support/core_ext/class/attribute"
|
|
5
|
+
|
|
6
|
+
require_relative "../locallingo"
|
|
7
|
+
require_relative "content_leaves"
|
|
8
|
+
|
|
9
|
+
module Locallingo
|
|
10
|
+
# Opt-in ActiveRecord concern (`require "locallingo/active_record"`) for the
|
|
11
|
+
# common shape: a JSON(B) column keyed by locale plus a JSON(B) state column
|
|
12
|
+
# holding one Drift entry per target locale. Supports ActiveRecord >= 7.1.
|
|
13
|
+
#
|
|
14
|
+
# class Article < ApplicationRecord
|
|
15
|
+
# include Locallingo::Translatable
|
|
16
|
+
#
|
|
17
|
+
# translates_content :body, # { "en" => { … }, "de" => { … } }
|
|
18
|
+
# state: :translation_state, # { "de" => { "source_hash", "target_hash", "manual" } }
|
|
19
|
+
# source_locale: -> { "en" }, # String/Symbol, or a callable run on the record
|
|
20
|
+
# target_locales: -> { %w[de sv] },
|
|
21
|
+
# translatable: ->(path, value) { true } # optional filter on flattened String leaves
|
|
22
|
+
# end
|
|
23
|
+
#
|
|
24
|
+
# A `before_save` callback (#mark_hand_edits) flags target locales a person
|
|
25
|
+
# changed as manual, so Locallingo::RecordTranslator never overwrites them.
|
|
26
|
+
module Translatable
|
|
27
|
+
extend ActiveSupport::Concern
|
|
28
|
+
|
|
29
|
+
Definition = Data.define(:column, :state, :source_locale, :target_locales, :translatable, :drift)
|
|
30
|
+
|
|
31
|
+
included do
|
|
32
|
+
class_attribute :locallingo_translatable, instance_writer: false
|
|
33
|
+
end
|
|
34
|
+
|
|
35
|
+
class_methods do
|
|
36
|
+
def translates_content(column, state:, source_locale:, target_locales:, translatable: nil, hasher: nil)
|
|
37
|
+
self.locallingo_translatable = Definition.new(
|
|
38
|
+
column: column.to_s, state: state.to_s, source_locale:, target_locales:,
|
|
39
|
+
translatable: translatable || ->(_path, _value) { true },
|
|
40
|
+
drift: hasher ? Drift.new(hasher:) : Drift.new
|
|
41
|
+
)
|
|
42
|
+
before_save :mark_hand_edits
|
|
43
|
+
end
|
|
44
|
+
end
|
|
45
|
+
|
|
46
|
+
def translation_source_locale = resolve_translation_option(locallingo_translatable.source_locale).to_s
|
|
47
|
+
|
|
48
|
+
def translation_target_locales
|
|
49
|
+
Array(resolve_translation_option(locallingo_translatable.target_locales)).map(&:to_s)
|
|
50
|
+
end
|
|
51
|
+
|
|
52
|
+
def translation_source = translation_content[translation_source_locale]
|
|
53
|
+
def translation_target(locale) = translation_content[locale.to_s]
|
|
54
|
+
def translation_entry(locale) = translation_states[locale.to_s]
|
|
55
|
+
def translation_source_digest = translation_drift.digest(translation_source)
|
|
56
|
+
|
|
57
|
+
def translation_status(locale)
|
|
58
|
+
translation_drift.status(source: translation_source, target: translation_target(locale),
|
|
59
|
+
entry: translation_entry(locale))
|
|
60
|
+
end
|
|
61
|
+
|
|
62
|
+
def translation_manual?(locale) = translation_drift.manual?(translation_entry(locale))
|
|
63
|
+
|
|
64
|
+
def translation_due?(locale, force: false, include_manual: false)
|
|
65
|
+
translation_drift.due?(source: translation_source, target: translation_target(locale),
|
|
66
|
+
entry: translation_entry(locale), force:, include_manual:)
|
|
67
|
+
end
|
|
68
|
+
|
|
69
|
+
# Records the current source and target of +locale+ as its state entry
|
|
70
|
+
# (assigned, not saved).
|
|
71
|
+
def record_translation(locale, manual: false)
|
|
72
|
+
entry = translation_drift.record(source: translation_source, target: translation_target(locale), manual:)
|
|
73
|
+
self[locallingo_translatable.state] = translation_states.merge(locale.to_s => entry)
|
|
74
|
+
end
|
|
75
|
+
|
|
76
|
+
# Flags every target locale a person changed as manual: a recorded target
|
|
77
|
+
# that no longer matches its target_hash, or an untracked locale written in
|
|
78
|
+
# this save. A cleared target is not flagged — it is missing and due again.
|
|
79
|
+
def mark_hand_edits
|
|
80
|
+
(translation_target_locales - [translation_source_locale]).each do |locale|
|
|
81
|
+
target = translation_target(locale)
|
|
82
|
+
next if translation_drift.blank?(target)
|
|
83
|
+
|
|
84
|
+
entry = translation_entry(locale)
|
|
85
|
+
next unless translation_drift.hand_edited?(target:, entry:) || (entry.nil? && translation_written?(locale))
|
|
86
|
+
|
|
87
|
+
# A recorded source_hash is kept: a hand edit does not make an
|
|
88
|
+
# outdated translation current.
|
|
89
|
+
manual = translation_drift.record(source: translation_source, target:, manual: true)
|
|
90
|
+
manual["source_hash"] = entry["source_hash"] if entry&.key?("source_hash")
|
|
91
|
+
self[locallingo_translatable.state] = translation_states.merge(locale => manual)
|
|
92
|
+
end
|
|
93
|
+
end
|
|
94
|
+
|
|
95
|
+
# The source's translatable String leaves, flattened (`steps[1].text`,
|
|
96
|
+
# `table[0][1]`). Unlike KeyFlattener, strings in nested arrays count.
|
|
97
|
+
def translation_segments
|
|
98
|
+
translation_leaves.filter_map do |path, (_, value)|
|
|
99
|
+
next unless value.is_a?(String) && !value.strip.empty?
|
|
100
|
+
|
|
101
|
+
[path, value] if locallingo_translatable.translatable.call(path, value)
|
|
102
|
+
end.to_h
|
|
103
|
+
end
|
|
104
|
+
|
|
105
|
+
# The target for translated +segments+ (`path => text`): a deep copy of the
|
|
106
|
+
# source with each translated leaf set in place, so every other value and
|
|
107
|
+
# container (numbers, nested lists, empty ones) keeps its place.
|
|
108
|
+
def build_translation_target(segments)
|
|
109
|
+
target = JSON.parse(JSON.generate(translation_source || {}))
|
|
110
|
+
leaves = translation_leaves
|
|
111
|
+
segments.each do |path, text|
|
|
112
|
+
*parents, last = leaves.fetch(path).first
|
|
113
|
+
(parents.empty? ? target : target.dig(*parents))[last] = text
|
|
114
|
+
end
|
|
115
|
+
target
|
|
116
|
+
end
|
|
117
|
+
|
|
118
|
+
# Column attributes that write +target+ for +locale+ plus its state entry.
|
|
119
|
+
def translation_attributes(locale, target, manual: false)
|
|
120
|
+
locale = locale.to_s
|
|
121
|
+
entry = translation_drift.record(source: translation_source, target:, manual:)
|
|
122
|
+
{
|
|
123
|
+
locallingo_translatable.column => translation_content.merge(locale => target),
|
|
124
|
+
locallingo_translatable.state => translation_states.merge(locale => entry)
|
|
125
|
+
}
|
|
126
|
+
end
|
|
127
|
+
|
|
128
|
+
def translation_drift = locallingo_translatable.drift
|
|
129
|
+
|
|
130
|
+
private
|
|
131
|
+
|
|
132
|
+
def translation_leaves = ContentLeaves.call(translation_source || {}, label: "#{self.class.name}##{id}")
|
|
133
|
+
|
|
134
|
+
def translation_content = self[locallingo_translatable.column] || {}
|
|
135
|
+
def translation_states = self[locallingo_translatable.state] || {}
|
|
136
|
+
|
|
137
|
+
def translation_written?(locale)
|
|
138
|
+
column = locallingo_translatable.column
|
|
139
|
+
return false unless will_save_change_to_attribute?(column)
|
|
140
|
+
|
|
141
|
+
(attribute_in_database(column) || {})[locale] != translation_target(locale)
|
|
142
|
+
end
|
|
143
|
+
|
|
144
|
+
def resolve_translation_option(option)
|
|
145
|
+
option.respond_to?(:call) ? instance_exec(&option) : option
|
|
146
|
+
end
|
|
147
|
+
end
|
|
148
|
+
end
|
|
@@ -0,0 +1,151 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "json"
|
|
4
|
+
|
|
5
|
+
module Locallingo
|
|
6
|
+
# Sends keys to the provider in batches and collects the translations.
|
|
7
|
+
#
|
|
8
|
+
# - Transport/API errors retry the same payload with backoff.
|
|
9
|
+
# - A reply that cannot be parsed is never resent as-is: one bad value (e.g.
|
|
10
|
+
# an unescaped quote) would fail it again. The batch is halved instead,
|
|
11
|
+
# down to single keys, so a bad value costs only its own key.
|
|
12
|
+
# - Keys the model omitted are re-batched for a few rounds; keys whose own
|
|
13
|
+
# one-key reply could not be parsed are not.
|
|
14
|
+
class BatchTranslator
|
|
15
|
+
MAX_RETRIES = 3
|
|
16
|
+
MAX_MISSING_RETRIES = 2
|
|
17
|
+
BASE_SLEEP_DURATION = 1.0
|
|
18
|
+
|
|
19
|
+
# `log` is called as `log.call(message, level: :info)`.
|
|
20
|
+
#
|
|
21
|
+
# A batch closes at `batch_size` keys or `max_batch_chars` characters of
|
|
22
|
+
# source text, whichever comes first; a single oversized value goes alone.
|
|
23
|
+
# `pause` is the sleep after each batch (0 in a background job).
|
|
24
|
+
def initialize(provider:, model:, batch_size:, log:, max_batch_chars: nil, pause: BASE_SLEEP_DURATION)
|
|
25
|
+
@provider = provider
|
|
26
|
+
@model = model
|
|
27
|
+
@batch_size = batch_size
|
|
28
|
+
@max_batch_chars = max_batch_chars
|
|
29
|
+
@pause = pause
|
|
30
|
+
@log = log
|
|
31
|
+
end
|
|
32
|
+
|
|
33
|
+
# Translates `keys` (looked up in `source`) and returns
|
|
34
|
+
# `[translations, failed_keys]`.
|
|
35
|
+
def call(source, keys, instructions:)
|
|
36
|
+
@source = source
|
|
37
|
+
@instructions = instructions
|
|
38
|
+
@unparseable = []
|
|
39
|
+
|
|
40
|
+
translated, missing = translate_keys(keys)
|
|
41
|
+
missing = retry_missing(translated, missing)
|
|
42
|
+
|
|
43
|
+
failed = missing + @unparseable
|
|
44
|
+
report_failed(failed)
|
|
45
|
+
[translated, failed]
|
|
46
|
+
end
|
|
47
|
+
|
|
48
|
+
private
|
|
49
|
+
|
|
50
|
+
def retry_missing(translated, missing)
|
|
51
|
+
missing -= @unparseable
|
|
52
|
+
round = 0
|
|
53
|
+
while missing.any? && round < MAX_MISSING_RETRIES
|
|
54
|
+
round += 1
|
|
55
|
+
log(" Retry round #{round}: #{missing.size} keys remaining...")
|
|
56
|
+
sleep(BASE_SLEEP_DURATION * (2**round))
|
|
57
|
+
retried, missing = translate_keys(missing)
|
|
58
|
+
translated.merge!(retried)
|
|
59
|
+
missing -= @unparseable
|
|
60
|
+
end
|
|
61
|
+
missing
|
|
62
|
+
end
|
|
63
|
+
|
|
64
|
+
def report_failed(failed)
|
|
65
|
+
return if failed.empty?
|
|
66
|
+
|
|
67
|
+
log(" WARNING: #{failed.size} keys failed after all retries:", level: :warn)
|
|
68
|
+
failed.each { |key| log(" - #{key}", level: :warn) }
|
|
69
|
+
end
|
|
70
|
+
|
|
71
|
+
def translate_keys(keys)
|
|
72
|
+
translations = {}
|
|
73
|
+
failed = []
|
|
74
|
+
|
|
75
|
+
batches(keys).each do |batch|
|
|
76
|
+
result = translate_batch(batch.to_h { |key| [key, @source[key]] })
|
|
77
|
+
batch.each do |key|
|
|
78
|
+
if result.key?(key) && !result[key].to_s.empty?
|
|
79
|
+
translations[key] = result[key]
|
|
80
|
+
else
|
|
81
|
+
failed << key
|
|
82
|
+
end
|
|
83
|
+
end
|
|
84
|
+
sleep(@pause) if @pause.positive?
|
|
85
|
+
end
|
|
86
|
+
|
|
87
|
+
[translations, failed]
|
|
88
|
+
end
|
|
89
|
+
|
|
90
|
+
def batches(keys)
|
|
91
|
+
return keys.each_slice(@batch_size).to_a unless @max_batch_chars
|
|
92
|
+
|
|
93
|
+
keys.each_with_object([[]]) do |key, groups|
|
|
94
|
+
groups << [] if full?(groups.last, key)
|
|
95
|
+
groups.last << key
|
|
96
|
+
end
|
|
97
|
+
end
|
|
98
|
+
|
|
99
|
+
def full?(batch, key)
|
|
100
|
+
return false if batch.empty?
|
|
101
|
+
|
|
102
|
+
batch.size >= @batch_size || chars(batch) + chars([key]) > @max_batch_chars
|
|
103
|
+
end
|
|
104
|
+
|
|
105
|
+
def chars(batch) = batch.sum { |key| @source[key].to_s.length }
|
|
106
|
+
|
|
107
|
+
def translate_batch(payload)
|
|
108
|
+
return {} if payload.empty?
|
|
109
|
+
|
|
110
|
+
request_batch(payload)
|
|
111
|
+
rescue JSON::ParserError => e
|
|
112
|
+
split_unparseable_batch(payload, e)
|
|
113
|
+
end
|
|
114
|
+
|
|
115
|
+
def split_unparseable_batch(payload, error)
|
|
116
|
+
if payload.size == 1
|
|
117
|
+
log(" Unparseable reply for #{payload.keys.first}: #{error.message}", level: :error)
|
|
118
|
+
@unparseable.concat(payload.keys)
|
|
119
|
+
return {}
|
|
120
|
+
end
|
|
121
|
+
|
|
122
|
+
log(" Unparseable reply for #{payload.size} keys, splitting the batch: #{error.message}", level: :warn)
|
|
123
|
+
payload.each_slice((payload.size / 2.0).ceil).with_object({}) do |half, result|
|
|
124
|
+
result.merge!(translate_batch(half.to_h))
|
|
125
|
+
end
|
|
126
|
+
end
|
|
127
|
+
|
|
128
|
+
def request_batch(payload)
|
|
129
|
+
retries = 0
|
|
130
|
+
begin
|
|
131
|
+
result = @provider.chat(model: @model, instructions: @instructions, payload:)
|
|
132
|
+
log(" Batch translated: #{result.keys.size}/#{payload.keys.size} keys")
|
|
133
|
+
result
|
|
134
|
+
rescue JSON::ParserError
|
|
135
|
+
raise
|
|
136
|
+
rescue StandardError => e
|
|
137
|
+
retries += 1
|
|
138
|
+
if retries < MAX_RETRIES
|
|
139
|
+
sleep_duration = BASE_SLEEP_DURATION * (2**retries)
|
|
140
|
+
log(" Batch failed (attempt #{retries}), retrying in #{sleep_duration}s: #{e.message}", level: :warn)
|
|
141
|
+
sleep(sleep_duration)
|
|
142
|
+
retry
|
|
143
|
+
end
|
|
144
|
+
log(" Translation batch failed after #{MAX_RETRIES} retries: #{e.message}", level: :error)
|
|
145
|
+
{}
|
|
146
|
+
end
|
|
147
|
+
end
|
|
148
|
+
|
|
149
|
+
def log(message, level: :info) = @log.call(message, level:)
|
|
150
|
+
end
|
|
151
|
+
end
|
data/lib/locallingo/cli.rb
CHANGED
|
@@ -99,17 +99,24 @@ module Locallingo
|
|
|
99
99
|
@options
|
|
100
100
|
end
|
|
101
101
|
|
|
102
|
+
def add_force_options(opts)
|
|
103
|
+
opts.on("-f", "--force", "Force re-translation of all keys") { @options[:force] = true }
|
|
104
|
+
opts.on("--force-key KEY", "Force re-translation of a specific key") do |v|
|
|
105
|
+
(@options[:force_keys] ||= []) << v
|
|
106
|
+
end
|
|
107
|
+
opts.on("--include-manual", "translate: let --force-key overwrite manual (hand-edited) keys") do
|
|
108
|
+
@options[:include_manual] = true
|
|
109
|
+
end
|
|
110
|
+
end
|
|
111
|
+
|
|
102
112
|
def build_parser
|
|
103
113
|
OptionParser.new do |opts|
|
|
104
114
|
opts.banner = "Usage: #{CLI_NAME} <command> [options]\n\nCommands:\n " \
|
|
105
115
|
"#{COMMANDS.keys.join(", ")}\n\nOptions:"
|
|
106
116
|
|
|
107
117
|
opts.on("-l", "--locale LOCALE", "Process specific locale") { |v| @options[:locale] = v }
|
|
108
|
-
opts
|
|
109
|
-
opts.on("--
|
|
110
|
-
(@options[:force_keys] ||= []) << v
|
|
111
|
-
end
|
|
112
|
-
opts.on("--key KEY", "accept-edits: accept a specific key (repeatable)") do |v|
|
|
118
|
+
add_force_options(opts)
|
|
119
|
+
opts.on("--key KEY", "accept-edits, quality: act on a specific key (repeatable)") do |v|
|
|
113
120
|
(@options[:keys] ||= []) << v
|
|
114
121
|
end
|
|
115
122
|
opts.on("--all", "accept-edits: mark every translated key as manual") { @options[:all] = true }
|
|
@@ -120,7 +127,7 @@ module Locallingo
|
|
|
120
127
|
@options[:strict] = true
|
|
121
128
|
@options[:strict_all] = true
|
|
122
129
|
end
|
|
123
|
-
opts
|
|
130
|
+
quality_options(opts)
|
|
124
131
|
opts.on("--json", "Output in JSON format") { @options[:format] = :json }
|
|
125
132
|
opts.on("--package PATH", "Scope to a package from .locallingo.yml") { |v| @options[:package] = v }
|
|
126
133
|
opts.on("-h", "--help", "Show this help") do
|
|
@@ -130,6 +137,13 @@ module Locallingo
|
|
|
130
137
|
end
|
|
131
138
|
end
|
|
132
139
|
|
|
140
|
+
def quality_options(opts)
|
|
141
|
+
opts.on("--ai", "Use AI for quality suggestions") { @options[:use_ai] = true }
|
|
142
|
+
opts.on("--prefix PREFIX", "quality: check only keys starting with PREFIX (e.g. yoga.)") do |v|
|
|
143
|
+
@options[:prefix] = v
|
|
144
|
+
end
|
|
145
|
+
end
|
|
146
|
+
|
|
133
147
|
def dispatch(command, config, options)
|
|
134
148
|
reporter = Reporter.new(config:, format: options[:format], cli_name: CLI_NAME)
|
|
135
149
|
|
|
@@ -155,13 +169,29 @@ module Locallingo
|
|
|
155
169
|
def cmd_translate(config, options)
|
|
156
170
|
mgr = manager(config, options)
|
|
157
171
|
puts "🔄 Translating..." if options[:verbose]
|
|
158
|
-
mgr.translate!(locale: options[:locale], force: options[:force],
|
|
172
|
+
failed = mgr.translate!(locale: options[:locale], force: options[:force],
|
|
173
|
+
force_keys: options[:force_keys] || [],
|
|
174
|
+
include_manual: options.fetch(:include_manual, false))
|
|
159
175
|
unless options[:dry_run]
|
|
160
176
|
puts "📝 Running post-translate hooks..."
|
|
161
177
|
mgr.run_after_translate_hooks
|
|
162
178
|
end
|
|
163
|
-
|
|
164
|
-
|
|
179
|
+
report_translated(failed, dry_run: options[:dry_run])
|
|
180
|
+
end
|
|
181
|
+
|
|
182
|
+
# Hooks have already run for the keys that were written; a failed key still
|
|
183
|
+
# fails the run (outside --dry-run) so CI does not ship missing translations.
|
|
184
|
+
def report_translated(failed, dry_run:)
|
|
185
|
+
count = failed.values.sum(&:size)
|
|
186
|
+
if count.zero?
|
|
187
|
+
puts "✅ Translation complete!"
|
|
188
|
+
else
|
|
189
|
+
puts "⚠️ Translation finished: #{count} #{count == 1 ? "key" : "keys"} failed " \
|
|
190
|
+
"(re-run with -v for details)"
|
|
191
|
+
failed.each { |locale, keys| keys.each { |key| puts " - #{locale}: #{key}" } }
|
|
192
|
+
end
|
|
193
|
+
puts "(dry run - no changes made)" if dry_run
|
|
194
|
+
exit 1 if count.positive? && !dry_run
|
|
165
195
|
end
|
|
166
196
|
|
|
167
197
|
def cmd_validate(config, options, reporter)
|
|
@@ -171,7 +201,11 @@ module Locallingo
|
|
|
171
201
|
|
|
172
202
|
def cmd_quality(config, options, reporter)
|
|
173
203
|
locale = options[:locale] || config.source_locale
|
|
174
|
-
|
|
204
|
+
raise Error, "--key and --prefix cannot be combined" if options[:keys] && options[:prefix]
|
|
205
|
+
|
|
206
|
+
suggestions = quality_checker(config, options).check(
|
|
207
|
+
locale:, use_ai: options[:use_ai], prefix: options[:prefix], keys: options[:keys]
|
|
208
|
+
)
|
|
175
209
|
reporter.quality(suggestions, locale:)
|
|
176
210
|
end
|
|
177
211
|
|
|
@@ -54,7 +54,10 @@ module Locallingo
|
|
|
54
54
|
def provider = data.fetch("provider").to_sym
|
|
55
55
|
def translate_model = dig("translate", "model")
|
|
56
56
|
def batch_size = dig("translate", "batch_size")
|
|
57
|
+
def max_batch_chars = dig("translate", "max_batch_chars")
|
|
58
|
+
def pause = dig("translate", "pause") || 1.0
|
|
57
59
|
def quality_model = dig("quality", "model")
|
|
60
|
+
def quality_sample_size = dig("quality", "sample_size")
|
|
58
61
|
def british_spellings? = !!dig("quality", "british_spellings")
|
|
59
62
|
def terminology_setting = dig("quality", "terminology")
|
|
60
63
|
|
|
@@ -99,7 +102,7 @@ module Locallingo
|
|
|
99
102
|
def after_translate = Array(data.fetch("after_translate", []))
|
|
100
103
|
|
|
101
104
|
BUILTIN_LANGUAGE_NAMES = {
|
|
102
|
-
"de" => "German", "sv" => "Swedish", "fr" => "French",
|
|
105
|
+
"en" => "English", "de" => "German", "sv" => "Swedish", "fr" => "French",
|
|
103
106
|
"af" => "Afrikaans", "es" => "Spanish", "it" => "Italian",
|
|
104
107
|
"nl" => "Dutch", "pt" => "Portuguese", "da" => "Danish",
|
|
105
108
|
"nb" => "Norwegian", "no" => "Norwegian", "fi" => "Finnish"
|