locallingo 0.4.3 → 0.5.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: e31de70770225e0fa02a19b214791252d80b79b46f69ff02fe4a4966079fb3bf
4
- data.tar.gz: 1b32c3ef52453c291676585fa0ca6a1da301879d292b98de13c017c6d03195e6
3
+ metadata.gz: b218e7422188db797e061b9cf1133b1b25a630e9e4c6237c04f0f7fa94dd2b4f
4
+ data.tar.gz: 742d0794ade8f32f9346d4a02b61554b7cb81321569c5297f9d93c3f1639f4e3
5
5
  SHA512:
6
- metadata.gz: c6293ed9127107f94800a144653e593aa2482aae0d5352dde2a7a02cd9043afedbcd441e112e3e3565e7842b981a9935623c70e94603a26fb0555e31b65397e9
7
- data.tar.gz: e20e64e38717897b90ab7607dfe032b52ecd3d9b12af37ec8615ca8811fe2d4fef6d3eefb6f00196e0b1d9f4eded368a0118f3dcbbddc2704d04f6ecb50106b3
6
+ metadata.gz: 48e0a2e2625a03fc0d78e9b5a9884efc4ec0ee7525533f150865d7bb391c473edd96f44bbca6974a24a6e564d4d08e644a0b36403a5c8a51b28a0943d54c1d1c
7
+ data.tar.gz: 4f08c9b89ab57351c5099ff6e31c4c24fa6b33a16df8ec0b70cb0ba365e292d90ef0fbe047d4b5494b9c2c7abe0627657c6ce4b545fd4a4552150f7338fd159d
data/CHANGELOG.md CHANGED
@@ -6,12 +6,81 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
6
6
 
7
7
  ## [Unreleased]
8
8
 
9
+ ### Added
10
+ - `Locallingo::Translator#translate(texts, from:, to:, style:, instructions:)`
11
+ translates a plain `{ key => text }` hash with the configured prompt,
12
+ glossary, language guide, provider and model, and returns `translations` and
13
+ `failed`. (#33)
14
+ - `Locallingo::Prompt.translation`: the translation prompt, public. `style:
15
+ :prose` keeps Markdown structure and drops the UI brevity rule;
16
+ `instructions:` is appended after the language guide. The CLI's `:ui`
17
+ prompt is unchanged. (#33)
18
+ - `Providers::RubyLLM.new(provider:, context:)` and `Translator.new(context:)`
19
+ take a `RubyLLM::Context`; with one, the global `RubyLLM.config` is never
20
+ written. (#33)
21
+ - `translate.max_batch_chars` limits the source characters per batch (a single
22
+ value over the limit is sent alone, never split); `translate.pause` sets the
23
+ sleep after each batch (default `1.0`). (#33)
24
+ - `Locallingo::Drift`: missing / outdated / manual / hand-edited decisions as a
25
+ value over the state entry shape, for strings and nested values, with a
26
+ pluggable hasher. `Manager` and the validators use it; behaviour is
27
+ unchanged. (#33)
28
+ - `require "locallingo/active_record"` adds `Locallingo::Translatable`
29
+ (`translates_content`), and `Locallingo::RecordTranslator` translates many
30
+ records in one batched call, writing each under its row lock. (#33)
31
+ - `lingo quality --prefix yoga.` checks only the keys under a prefix, in both
32
+ the static checks and the AI pass. A prefix that matches nothing exits `1`. (#10, #29)
33
+ - `lingo quality --key a.b` (repeatable) checks exactly the named keys, and the
34
+ AI pass reviews every named key without sampling. An unknown key exits `1`;
35
+ `--key` and `--prefix` cannot be combined. (#10, #29)
36
+ - `quality.sample_size` (default `100`) sets how many keys `quality --ai`
37
+ samples. (#10, #29)
38
+ - `lingo translate --include-manual` lets `--force-key` overwrite keys flagged
39
+ `manual`; each overwritten key is named on stderr and its `manual` flag
40
+ survives. (#11, #30)
41
+
9
42
  ### Changed
10
- - Allow `ruby_llm` 2.x (dependency constraint widened from `< 2` to `< 3`). The
11
- gem only uses RubyLLM's stable chat API and guarded config accessors, so a
12
- host app can take the RubyLLM 2.0 bump without waiting on Locallingo.
43
+ - `lingo translate --force-key` skips keys flagged `manual` (hand-edited, via
44
+ `accept-edits`) and names each skipped key on stderr, the same protection
45
+ `--force` already gave. If every named key is manual, nothing is translated.
46
+ Pass `--include-manual` to overwrite them as before. (#11, #30)
47
+ - The translation prompt asks for natural, idiomatic text in the target
48
+ language ("into natural, idiomatic German") instead of a word-for-word
49
+ translation: the meaning, phrasing and UI conventions a native speaker would
50
+ use, with idioms replaced by their local equivalent. Only newly translated or
51
+ outdated keys are affected; run `lingo translate --force` to redo all
52
+ non-manual keys (hand-edited `manual` keys stay untouched — redo those one at
53
+ a time with `--force-key <key> --include-manual` and compare).
54
+ The source language is now named too (`en` → "English").
55
+ - The translation prompt no longer hardcodes "Use formal business language".
56
+ Register (formal "Sie"/"Lei" vs. informal "du"/"tu") belongs in each
57
+ locale's `language_guides` entry; add it there to keep a formal tone.
58
+ - `lingo translate` exits `1` when any key fails to translate, and prints
59
+ `⚠️ Translation finished: N keys failed` with the failed keys instead of
60
+ `✅ Translation complete!`. Translated keys are still written and the
61
+ `after_translate` hooks still run; `--dry-run` reports failures but exits `0`.
62
+ Appending `|| true` restores the old always-`0` behaviour, but also hides
63
+ every other failure (missing credentials, a broken config). (#25)
64
+ - `Manager#translate!` returns `{ locale => failed_keys }`. (#25)
65
+ - Require `ruby_llm` >= 2.1 (constraint is now `>= 2.1, < 3`; host apps pinned
66
+ to RubyLLM 1.x or 2.0 must bump). The translation and quality-review system
67
+ prompts are marked as prompt cache boundaries; the translation prompt is
68
+ reused by every batch of a locale, so providers with prompt caching
69
+ (Anthropic, OpenAI-compatible) bill that repeated prefix at the cached rate. Anthropic only caches prompts
70
+ above its minimum length (~1024 tokens); shorter prompts are sent uncached.
13
71
 
14
72
  ### Fixed
73
+ - `lingo quality --ai` includes the locale's `language_guides` entry in the
74
+ review prompt, as `translate` does, so the AI pass can flag violations of the
75
+ guide (e.g. du/Sie register). (#9, #29)
76
+ - One invalid value in a model reply (e.g. typographic quotes returned as
77
+ unescaped `"`) no longer fails every key in its batch. A reply that cannot be
78
+ parsed is never resent as-is: the batch is split in half, down to single keys,
79
+ so only the bad key fails. Transport errors still retry with backoff. (#25)
80
+ - When a ```json fence holds invalid JSON, the parse error now names the line
81
+ and column inside the fence instead of the fence at line 1. (#25)
82
+ - The translation prompt asks the model to escape `"` inside values as `\"`
83
+ and keep typographic quotes (“ ” „ ‚ « ») as they appear in the source. (#25)
15
84
  - State files are written with a trailing final newline, so the output is
16
85
  POSIX-conformant and files that already end with a newline are no longer
17
86
  rewritten with newline-only diffs. One-time effect after upgrading: each
@@ -21,6 +90,8 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
21
90
  through `translate` or `accept-edits`) get a baseline and the `manual_edits`
22
91
  validator can watch them. An existing `target_hash` is still never
23
92
  recomputed — that would silently absorb hand-edit drift.
93
+ - The `duplicate_values` validator reports findings under the configured
94
+ `source_locale` instead of a hard-coded `en`. (#15)
24
95
 
25
96
  ### Fixed (0.4.0)
26
97
  - `lingo sync` no longer destroys hand-edit protection: it now only refreshes
data/README.md CHANGED
@@ -100,9 +100,11 @@ With no `packages:`, the whole app uses `defaults` — the common case.
100
100
  lingo status # translation status per locale
101
101
  lingo translate --locale de # translate missing/changed keys
102
102
  lingo translate --force # re-translate everything
103
+ lingo translate --force-key a.b.c # re-translate one key (skips manual; add --include-manual)
103
104
  lingo validate --strict # CI gate (exit 1 on strict-tier issues)
104
105
  lingo validate --strict-all # stricter CI gate
105
106
  lingo quality --ai # quality linting (+ optional AI pass)
107
+ lingo quality --ai --prefix yoga. # lint one namespace
106
108
  lingo fix-quality --locale en # auto-fix fixable issues
107
109
  lingo accept-edits --locale de # protect hand-edited translations (drifted keys only)
108
110
  lingo accept-edits --key a.b.c # protect a specific key
@@ -144,6 +146,98 @@ Locallingo::Manager.new(config:).validate # => [violations]
144
146
  Locallingo::QualityChecker.new(config:).check(locale: "en")
145
147
  ```
146
148
 
149
+ ### Translating text that is not in locale files
150
+
151
+ `Locallingo::Translator` runs the same prompt (rules, glossary, language guide),
152
+ provider, model and batching over a plain `{ key => text }` hash:
153
+
154
+ ```ruby
155
+ require "ruby_llm" # for RubyLLM.context (Bundler.require already loads it in Rails)
156
+
157
+ translator = Locallingo::Translator.new(
158
+ config: Locallingo.configuration(root_path: Dir.pwd),
159
+ context: RubyLLM.context { |c| c.anthropic_api_key = tenant_key }, # optional, per-tenant key
160
+ pause: 0 # no sleep between batches
161
+ )
162
+
163
+ result = translator.translate(
164
+ { "title" => "Hello", "intro" => "Some **markdown**." },
165
+ from: "en", to: "de",
166
+ style: :prose, # :ui (default) or :prose (keeps Markdown, no brevity rule)
167
+ instructions: "Write in a warm, plain voice." # appended after the language guide
168
+ )
169
+ result.translations # => { "title" => "Hallo", "intro" => "Etwas **Markdown**." }
170
+ result.failed # => keys that failed after retries
171
+ ```
172
+
173
+ - With a `context:`, every call uses that `RubyLLM::Context`'s config; the
174
+ global `RubyLLM.config` is never written, so tenants can translate
175
+ concurrently.
176
+ - `MissingCredentialsError` is raised before any call when no key is found.
177
+ - `translate.max_batch_chars` (or `max_batch_chars:`) limits the source
178
+ characters per batch (a single longer value is sent alone, not split); `translate.pause` (or `pause:`) is the sleep after each
179
+ batch. Retries of failed or omitted keys keep their own backoff.
180
+ - `Locallingo::Prompt.translation(config:, from:, to:, style:, instructions:)`
181
+ returns the prompt itself.
182
+
183
+ `Locallingo::Drift` is the missing / outdated / hand-edited rule over the state
184
+ entry shape (`source_hash`, `target_hash`, `manual`), for strings or nested
185
+ values:
186
+
187
+ ```ruby
188
+ drift = Locallingo::Drift.new # hasher: any callable(String) -> String
189
+ drift.status(source:, target:, entry:) # => :missing | :outdated | :current
190
+ drift.due?(source:, target:, entry:, force: false, include_manual: false)
191
+ drift.hand_edited?(target:, entry:)
192
+ drift.record(source:, target:, manual: false) # => a new entry
193
+ ```
194
+
195
+ ### Translating records
196
+
197
+ `require "locallingo/active_record"` (ActiveRecord >= 7.1; nothing loads
198
+ ActiveRecord otherwise) adds a concern for a JSON(B) column keyed by locale
199
+ plus a JSON(B) state column:
200
+
201
+ ```ruby
202
+ class Article < ApplicationRecord
203
+ include Locallingo::Translatable
204
+
205
+ translates_content :body, # { "en" => { … }, "de" => { … } }
206
+ state: :translation_state, # { "de" => { "source_hash", "target_hash", "manual" } }
207
+ source_locale: -> { "en" }, # String/Symbol, or a callable run on the record
208
+ target_locales: -> { %w[de sv] },
209
+ translatable: ->(path, value) { true } # optional filter on flattened String leaves
210
+ end
211
+
212
+ article.translation_status(:de) # => :missing | :outdated | :current
213
+ article.translation_manual?(:de)
214
+ article.translation_due?(:de, force: false)
215
+ article.record_translation(:de, manual: false)
216
+
217
+ Locallingo::RecordTranslator.new(translator:)
218
+ .call(Article.where(id: ids), to: "de", style: :prose)
219
+ # => { translated: 3, failed: { article_id => [paths] }, skipped: [ids] }
220
+ ```
221
+
222
+ - Saving a record runs `mark_hand_edits`: a target locale a person changed is
223
+ flagged `manual` and never machine-overwritten (`include_manual: true`
224
+ overrides, keeping the flag). That includes a target locale saved for the
225
+ first time, e.g. a source copy used as a placeholder at `create!`: leave
226
+ untranslated locales out of the column instead. A cleared locale is not
227
+ flagged and is due again, unless it was already `manual`: the flag stays, so
228
+ retranslating it needs `include_manual: true`.
229
+ - `RecordTranslator` sends one batched translation per source locale across all
230
+ due records, then writes each record under `record.with_lock`. A record whose
231
+ source changed, or whose target became manual, while the model answered is
232
+ skipped (`skipped:`) and left for the next run, as is a due record with no
233
+ translatable text.
234
+ - Nested values keep their shape; leaves the `translatable:` filter rejects are
235
+ copied from the source. A record with any segment that failed after retries
236
+ is not written at all: its paths are reported in `failed:`, and it stays due
237
+ for the next run (your fallback to the source locale applies meanwhile).
238
+ - Writes use `update_columns`; pass `write: ->(record, attributes) { … }` to
239
+ bump a lock version or run validations.
240
+
147
241
  ## License
148
242
 
149
243
  MIT © Mikael Henriksson
@@ -26,9 +26,18 @@ defaults:
26
26
  translate:
27
27
  model: gpt-5-mini
28
28
  batch_size: 20
29
+ # Optional cap on the characters of source text per batch (a batch closes at
30
+ # batch_size keys or max_batch_chars, whichever comes first). Useful for long
31
+ # prose; a single oversized value is still sent alone. null = keys only.
32
+ max_batch_chars: null
33
+ # Seconds to sleep after each batch (rate-limit courtesy). 0 disables it.
34
+ pause: 1.0
29
35
 
30
36
  quality:
31
37
  model: gpt-4o-mini
38
+ # How many randomly sampled keys `quality --ai` sends for review per run
39
+ # (`--key` reviews every named key instead).
40
+ sample_size: 100
32
41
  # Flag en-only British-vs-American spelling drift (organization -> organisation).
33
42
  british_spellings: false
34
43
  # Named built-in terminology list ("business" | "banking" | "none"), or a
@@ -41,7 +50,8 @@ defaults:
41
50
  context: "a business application"
42
51
  placeholder_style: "%<name>s, %<count>s"
43
52
  glossary: {}
44
- # Extra per-locale style guidance appended to the translation prompt. Value is
53
+ # Extra per-locale style guidance appended to the translation and AI quality
54
+ # review prompts. Value is
45
55
  # inline text or `file: path/to/guide.md`.
46
56
  language_guides: {}
47
57
 
@@ -0,0 +1,148 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "active_support/concern"
4
+ require "active_support/core_ext/class/attribute"
5
+
6
+ require_relative "../locallingo"
7
+ require_relative "content_leaves"
8
+
9
+ module Locallingo
10
+ # Opt-in ActiveRecord concern (`require "locallingo/active_record"`) for the
11
+ # common shape: a JSON(B) column keyed by locale plus a JSON(B) state column
12
+ # holding one Drift entry per target locale. Supports ActiveRecord >= 7.1.
13
+ #
14
+ # class Article < ApplicationRecord
15
+ # include Locallingo::Translatable
16
+ #
17
+ # translates_content :body, # { "en" => { … }, "de" => { … } }
18
+ # state: :translation_state, # { "de" => { "source_hash", "target_hash", "manual" } }
19
+ # source_locale: -> { "en" }, # String/Symbol, or a callable run on the record
20
+ # target_locales: -> { %w[de sv] },
21
+ # translatable: ->(path, value) { true } # optional filter on flattened String leaves
22
+ # end
23
+ #
24
+ # A `before_save` callback (#mark_hand_edits) flags target locales a person
25
+ # changed as manual, so Locallingo::RecordTranslator never overwrites them.
26
+ module Translatable
27
+ extend ActiveSupport::Concern
28
+
29
+ Definition = Data.define(:column, :state, :source_locale, :target_locales, :translatable, :drift)
30
+
31
+ included do
32
+ class_attribute :locallingo_translatable, instance_writer: false
33
+ end
34
+
35
+ class_methods do
36
+ def translates_content(column, state:, source_locale:, target_locales:, translatable: nil, hasher: nil)
37
+ self.locallingo_translatable = Definition.new(
38
+ column: column.to_s, state: state.to_s, source_locale:, target_locales:,
39
+ translatable: translatable || ->(_path, _value) { true },
40
+ drift: hasher ? Drift.new(hasher:) : Drift.new
41
+ )
42
+ before_save :mark_hand_edits
43
+ end
44
+ end
45
+
46
+ def translation_source_locale = resolve_translation_option(locallingo_translatable.source_locale).to_s
47
+
48
+ def translation_target_locales
49
+ Array(resolve_translation_option(locallingo_translatable.target_locales)).map(&:to_s)
50
+ end
51
+
52
+ def translation_source = translation_content[translation_source_locale]
53
+ def translation_target(locale) = translation_content[locale.to_s]
54
+ def translation_entry(locale) = translation_states[locale.to_s]
55
+ def translation_source_digest = translation_drift.digest(translation_source)
56
+
57
+ def translation_status(locale)
58
+ translation_drift.status(source: translation_source, target: translation_target(locale),
59
+ entry: translation_entry(locale))
60
+ end
61
+
62
+ def translation_manual?(locale) = translation_drift.manual?(translation_entry(locale))
63
+
64
+ def translation_due?(locale, force: false, include_manual: false)
65
+ translation_drift.due?(source: translation_source, target: translation_target(locale),
66
+ entry: translation_entry(locale), force:, include_manual:)
67
+ end
68
+
69
+ # Records the current source and target of +locale+ as its state entry
70
+ # (assigned, not saved).
71
+ def record_translation(locale, manual: false)
72
+ entry = translation_drift.record(source: translation_source, target: translation_target(locale), manual:)
73
+ self[locallingo_translatable.state] = translation_states.merge(locale.to_s => entry)
74
+ end
75
+
76
+ # Flags every target locale a person changed as manual: a recorded target
77
+ # that no longer matches its target_hash, or an untracked locale written in
78
+ # this save. A cleared target is not flagged — it is missing and due again.
79
+ def mark_hand_edits
80
+ (translation_target_locales - [translation_source_locale]).each do |locale|
81
+ target = translation_target(locale)
82
+ next if translation_drift.blank?(target)
83
+
84
+ entry = translation_entry(locale)
85
+ next unless translation_drift.hand_edited?(target:, entry:) || (entry.nil? && translation_written?(locale))
86
+
87
+ # A recorded source_hash is kept: a hand edit does not make an
88
+ # outdated translation current.
89
+ manual = translation_drift.record(source: translation_source, target:, manual: true)
90
+ manual["source_hash"] = entry["source_hash"] if entry&.key?("source_hash")
91
+ self[locallingo_translatable.state] = translation_states.merge(locale => manual)
92
+ end
93
+ end
94
+
95
+ # The source's translatable String leaves, flattened (`steps[1].text`,
96
+ # `table[0][1]`). Unlike KeyFlattener, strings in nested arrays count.
97
+ def translation_segments
98
+ translation_leaves.filter_map do |path, (_, value)|
99
+ next unless value.is_a?(String) && !value.strip.empty?
100
+
101
+ [path, value] if locallingo_translatable.translatable.call(path, value)
102
+ end.to_h
103
+ end
104
+
105
+ # The target for translated +segments+ (`path => text`): a deep copy of the
106
+ # source with each translated leaf set in place, so every other value and
107
+ # container (numbers, nested lists, empty ones) keeps its place.
108
+ def build_translation_target(segments)
109
+ target = JSON.parse(JSON.generate(translation_source || {}))
110
+ leaves = translation_leaves
111
+ segments.each do |path, text|
112
+ *parents, last = leaves.fetch(path).first
113
+ (parents.empty? ? target : target.dig(*parents))[last] = text
114
+ end
115
+ target
116
+ end
117
+
118
+ # Column attributes that write +target+ for +locale+ plus its state entry.
119
+ def translation_attributes(locale, target, manual: false)
120
+ locale = locale.to_s
121
+ entry = translation_drift.record(source: translation_source, target:, manual:)
122
+ {
123
+ locallingo_translatable.column => translation_content.merge(locale => target),
124
+ locallingo_translatable.state => translation_states.merge(locale => entry)
125
+ }
126
+ end
127
+
128
+ def translation_drift = locallingo_translatable.drift
129
+
130
+ private
131
+
132
+ def translation_leaves = ContentLeaves.call(translation_source || {}, label: "#{self.class.name}##{id}")
133
+
134
+ def translation_content = self[locallingo_translatable.column] || {}
135
+ def translation_states = self[locallingo_translatable.state] || {}
136
+
137
+ def translation_written?(locale)
138
+ column = locallingo_translatable.column
139
+ return false unless will_save_change_to_attribute?(column)
140
+
141
+ (attribute_in_database(column) || {})[locale] != translation_target(locale)
142
+ end
143
+
144
+ def resolve_translation_option(option)
145
+ option.respond_to?(:call) ? instance_exec(&option) : option
146
+ end
147
+ end
148
+ end
@@ -0,0 +1,151 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "json"
4
+
5
+ module Locallingo
6
+ # Sends keys to the provider in batches and collects the translations.
7
+ #
8
+ # - Transport/API errors retry the same payload with backoff.
9
+ # - A reply that cannot be parsed is never resent as-is: one bad value (e.g.
10
+ # an unescaped quote) would fail it again. The batch is halved instead,
11
+ # down to single keys, so a bad value costs only its own key.
12
+ # - Keys the model omitted are re-batched for a few rounds; keys whose own
13
+ # one-key reply could not be parsed are not.
14
+ class BatchTranslator
15
+ MAX_RETRIES = 3
16
+ MAX_MISSING_RETRIES = 2
17
+ BASE_SLEEP_DURATION = 1.0
18
+
19
+ # `log` is called as `log.call(message, level: :info)`.
20
+ #
21
+ # A batch closes at `batch_size` keys or `max_batch_chars` characters of
22
+ # source text, whichever comes first; a single oversized value goes alone.
23
+ # `pause` is the sleep after each batch (0 in a background job).
24
+ def initialize(provider:, model:, batch_size:, log:, max_batch_chars: nil, pause: BASE_SLEEP_DURATION)
25
+ @provider = provider
26
+ @model = model
27
+ @batch_size = batch_size
28
+ @max_batch_chars = max_batch_chars
29
+ @pause = pause
30
+ @log = log
31
+ end
32
+
33
+ # Translates `keys` (looked up in `source`) and returns
34
+ # `[translations, failed_keys]`.
35
+ def call(source, keys, instructions:)
36
+ @source = source
37
+ @instructions = instructions
38
+ @unparseable = []
39
+
40
+ translated, missing = translate_keys(keys)
41
+ missing = retry_missing(translated, missing)
42
+
43
+ failed = missing + @unparseable
44
+ report_failed(failed)
45
+ [translated, failed]
46
+ end
47
+
48
+ private
49
+
50
+ def retry_missing(translated, missing)
51
+ missing -= @unparseable
52
+ round = 0
53
+ while missing.any? && round < MAX_MISSING_RETRIES
54
+ round += 1
55
+ log(" Retry round #{round}: #{missing.size} keys remaining...")
56
+ sleep(BASE_SLEEP_DURATION * (2**round))
57
+ retried, missing = translate_keys(missing)
58
+ translated.merge!(retried)
59
+ missing -= @unparseable
60
+ end
61
+ missing
62
+ end
63
+
64
+ def report_failed(failed)
65
+ return if failed.empty?
66
+
67
+ log(" WARNING: #{failed.size} keys failed after all retries:", level: :warn)
68
+ failed.each { |key| log(" - #{key}", level: :warn) }
69
+ end
70
+
71
+ def translate_keys(keys)
72
+ translations = {}
73
+ failed = []
74
+
75
+ batches(keys).each do |batch|
76
+ result = translate_batch(batch.to_h { |key| [key, @source[key]] })
77
+ batch.each do |key|
78
+ if result.key?(key) && !result[key].to_s.empty?
79
+ translations[key] = result[key]
80
+ else
81
+ failed << key
82
+ end
83
+ end
84
+ sleep(@pause) if @pause.positive?
85
+ end
86
+
87
+ [translations, failed]
88
+ end
89
+
90
+ def batches(keys)
91
+ return keys.each_slice(@batch_size).to_a unless @max_batch_chars
92
+
93
+ keys.each_with_object([[]]) do |key, groups|
94
+ groups << [] if full?(groups.last, key)
95
+ groups.last << key
96
+ end
97
+ end
98
+
99
+ def full?(batch, key)
100
+ return false if batch.empty?
101
+
102
+ batch.size >= @batch_size || chars(batch) + chars([key]) > @max_batch_chars
103
+ end
104
+
105
+ def chars(batch) = batch.sum { |key| @source[key].to_s.length }
106
+
107
+ def translate_batch(payload)
108
+ return {} if payload.empty?
109
+
110
+ request_batch(payload)
111
+ rescue JSON::ParserError => e
112
+ split_unparseable_batch(payload, e)
113
+ end
114
+
115
+ def split_unparseable_batch(payload, error)
116
+ if payload.size == 1
117
+ log(" Unparseable reply for #{payload.keys.first}: #{error.message}", level: :error)
118
+ @unparseable.concat(payload.keys)
119
+ return {}
120
+ end
121
+
122
+ log(" Unparseable reply for #{payload.size} keys, splitting the batch: #{error.message}", level: :warn)
123
+ payload.each_slice((payload.size / 2.0).ceil).with_object({}) do |half, result|
124
+ result.merge!(translate_batch(half.to_h))
125
+ end
126
+ end
127
+
128
+ def request_batch(payload)
129
+ retries = 0
130
+ begin
131
+ result = @provider.chat(model: @model, instructions: @instructions, payload:)
132
+ log(" Batch translated: #{result.keys.size}/#{payload.keys.size} keys")
133
+ result
134
+ rescue JSON::ParserError
135
+ raise
136
+ rescue StandardError => e
137
+ retries += 1
138
+ if retries < MAX_RETRIES
139
+ sleep_duration = BASE_SLEEP_DURATION * (2**retries)
140
+ log(" Batch failed (attempt #{retries}), retrying in #{sleep_duration}s: #{e.message}", level: :warn)
141
+ sleep(sleep_duration)
142
+ retry
143
+ end
144
+ log(" Translation batch failed after #{MAX_RETRIES} retries: #{e.message}", level: :error)
145
+ {}
146
+ end
147
+ end
148
+
149
+ def log(message, level: :info) = @log.call(message, level:)
150
+ end
151
+ end
@@ -99,17 +99,24 @@ module Locallingo
99
99
  @options
100
100
  end
101
101
 
102
+ def add_force_options(opts)
103
+ opts.on("-f", "--force", "Force re-translation of all keys") { @options[:force] = true }
104
+ opts.on("--force-key KEY", "Force re-translation of a specific key") do |v|
105
+ (@options[:force_keys] ||= []) << v
106
+ end
107
+ opts.on("--include-manual", "translate: let --force-key overwrite manual (hand-edited) keys") do
108
+ @options[:include_manual] = true
109
+ end
110
+ end
111
+
102
112
  def build_parser
103
113
  OptionParser.new do |opts|
104
114
  opts.banner = "Usage: #{CLI_NAME} <command> [options]\n\nCommands:\n " \
105
115
  "#{COMMANDS.keys.join(", ")}\n\nOptions:"
106
116
 
107
117
  opts.on("-l", "--locale LOCALE", "Process specific locale") { |v| @options[:locale] = v }
108
- opts.on("-f", "--force", "Force re-translation of all keys") { @options[:force] = true }
109
- opts.on("--force-key KEY", "Force re-translation of a specific key") do |v|
110
- (@options[:force_keys] ||= []) << v
111
- end
112
- opts.on("--key KEY", "accept-edits: accept a specific key (repeatable)") do |v|
118
+ add_force_options(opts)
119
+ opts.on("--key KEY", "accept-edits, quality: act on a specific key (repeatable)") do |v|
113
120
  (@options[:keys] ||= []) << v
114
121
  end
115
122
  opts.on("--all", "accept-edits: mark every translated key as manual") { @options[:all] = true }
@@ -120,7 +127,7 @@ module Locallingo
120
127
  @options[:strict] = true
121
128
  @options[:strict_all] = true
122
129
  end
123
- opts.on("--ai", "Use AI for quality suggestions") { @options[:use_ai] = true }
130
+ quality_options(opts)
124
131
  opts.on("--json", "Output in JSON format") { @options[:format] = :json }
125
132
  opts.on("--package PATH", "Scope to a package from .locallingo.yml") { |v| @options[:package] = v }
126
133
  opts.on("-h", "--help", "Show this help") do
@@ -130,6 +137,13 @@ module Locallingo
130
137
  end
131
138
  end
132
139
 
140
+ def quality_options(opts)
141
+ opts.on("--ai", "Use AI for quality suggestions") { @options[:use_ai] = true }
142
+ opts.on("--prefix PREFIX", "quality: check only keys starting with PREFIX (e.g. yoga.)") do |v|
143
+ @options[:prefix] = v
144
+ end
145
+ end
146
+
133
147
  def dispatch(command, config, options)
134
148
  reporter = Reporter.new(config:, format: options[:format], cli_name: CLI_NAME)
135
149
 
@@ -155,13 +169,29 @@ module Locallingo
155
169
  def cmd_translate(config, options)
156
170
  mgr = manager(config, options)
157
171
  puts "🔄 Translating..." if options[:verbose]
158
- mgr.translate!(locale: options[:locale], force: options[:force], force_keys: options[:force_keys] || [])
172
+ failed = mgr.translate!(locale: options[:locale], force: options[:force],
173
+ force_keys: options[:force_keys] || [],
174
+ include_manual: options.fetch(:include_manual, false))
159
175
  unless options[:dry_run]
160
176
  puts "📝 Running post-translate hooks..."
161
177
  mgr.run_after_translate_hooks
162
178
  end
163
- puts "✅ Translation complete!"
164
- puts "(dry run - no changes made)" if options[:dry_run]
179
+ report_translated(failed, dry_run: options[:dry_run])
180
+ end
181
+
182
+ # Hooks have already run for the keys that were written; a failed key still
183
+ # fails the run (outside --dry-run) so CI does not ship missing translations.
184
+ def report_translated(failed, dry_run:)
185
+ count = failed.values.sum(&:size)
186
+ if count.zero?
187
+ puts "✅ Translation complete!"
188
+ else
189
+ puts "⚠️ Translation finished: #{count} #{count == 1 ? "key" : "keys"} failed " \
190
+ "(re-run with -v for details)"
191
+ failed.each { |locale, keys| keys.each { |key| puts " - #{locale}: #{key}" } }
192
+ end
193
+ puts "(dry run - no changes made)" if dry_run
194
+ exit 1 if count.positive? && !dry_run
165
195
  end
166
196
 
167
197
  def cmd_validate(config, options, reporter)
@@ -171,7 +201,11 @@ module Locallingo
171
201
 
172
202
  def cmd_quality(config, options, reporter)
173
203
  locale = options[:locale] || config.source_locale
174
- suggestions = quality_checker(config, options).check(locale:, use_ai: options[:use_ai])
204
+ raise Error, "--key and --prefix cannot be combined" if options[:keys] && options[:prefix]
205
+
206
+ suggestions = quality_checker(config, options).check(
207
+ locale:, use_ai: options[:use_ai], prefix: options[:prefix], keys: options[:keys]
208
+ )
175
209
  reporter.quality(suggestions, locale:)
176
210
  end
177
211
 
@@ -54,7 +54,10 @@ module Locallingo
54
54
  def provider = data.fetch("provider").to_sym
55
55
  def translate_model = dig("translate", "model")
56
56
  def batch_size = dig("translate", "batch_size")
57
+ def max_batch_chars = dig("translate", "max_batch_chars")
58
+ def pause = dig("translate", "pause") || 1.0
57
59
  def quality_model = dig("quality", "model")
60
+ def quality_sample_size = dig("quality", "sample_size")
58
61
  def british_spellings? = !!dig("quality", "british_spellings")
59
62
  def terminology_setting = dig("quality", "terminology")
60
63
 
@@ -99,7 +102,7 @@ module Locallingo
99
102
  def after_translate = Array(data.fetch("after_translate", []))
100
103
 
101
104
  BUILTIN_LANGUAGE_NAMES = {
102
- "de" => "German", "sv" => "Swedish", "fr" => "French",
105
+ "en" => "English", "de" => "German", "sv" => "Swedish", "fr" => "French",
103
106
  "af" => "Afrikaans", "es" => "Spanish", "it" => "Italian",
104
107
  "nl" => "Dutch", "pt" => "Portuguese", "da" => "Danish",
105
108
  "nb" => "Norwegian", "no" => "Norwegian", "fi" => "Finnish"