locallingo 0.6.0 → 0.6.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 47b5cfaffa1567e536de600f6f963538606fc386a89b61e90ad9cc4dd8327132
4
- data.tar.gz: 767fa1b9c49c66a0b5b5e1bd6140edbca50edade201f415c9bfde2c6f0448206
3
+ metadata.gz: '08b89ac361ed67239fe3e8574396cf5ffb1b0fb20a04385cea02d350b608d503'
4
+ data.tar.gz: 2de02e7a1ca34d339bff982427a411bb299c02907d3d8545e8dcb1092a747031
5
5
  SHA512:
6
- metadata.gz: fad2120d8032a4a019dda5f6c82cd2f597de9dd79f9bb4eab1b9026530c859316c232314325398f51e12d61fc043c95882960969d5be325a80e6676463b55862
7
- data.tar.gz: ff18516ac297ff0612111d0326552e6e771a6e0299854c598f3ea5645e9baaf722dd80e2571fc68186f3225af3eb35a69dc496107904feedd2cfc9d7356b1e8a
6
+ metadata.gz: b488c4209a18715d8772e2616b2bf80eeafe0973d0a97be9fc5da8b01417308550aac86fc3b512fec3cd480964aac0281e02a126319a882aaf4fb49776748bae
7
+ data.tar.gz: b4c13e6f0ca99cb0a0a54b1d56be91dad3b5810cf40f771ac653e63ec2f2ab3bf24fa460e1ecb661c7ac280f0ef4961df1a9b97070a991f8fce6c39244b14666
data/CHANGELOG.md CHANGED
@@ -4,7 +4,25 @@ All notable changes to this project are documented here. The format is based on
4
4
  [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project
5
5
  adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
6
6
 
7
- ## [0.6.0]
7
+ ## [Unreleased]
8
+
9
+ ### Added
10
+ - `Translator#translate(…, placeholders:)` takes a list of Regexps (default:
11
+ the new `translate.placeholders` config key, regex strings, `[]`) for the
12
+ tokens the texts use. The prompt names the patterns, and a value whose
13
+ tokens differ from its source's (exact text, by count) is re-requested in
14
+ the omitted-key rounds, then reported in `failed` instead of returned.
15
+ `Locallingo::Placeholders` is the check. (#43)
16
+ - `Translator::Result#rejections` (`{ key => { "missing" => [...], "extra" =>
17
+ [...] } }`) and `BatchTranslator.new(…, accept:)`, whose `call` now returns
18
+ `[translations, failed, rejections]`. `RecordTranslator#call` passes
19
+ `placeholders:` through. The `lingo` CLI is unchanged. (#43)
20
+ - `QualityChecker#check_texts(texts, locale:, use_ai: false)` runs the static
21
+ checks, terminology, British spellings and the optional AI review over a
22
+ `{ key => text }` hash instead of the locale files; `QualityChecker.new`
23
+ takes `context:` (a `RubyLLM::Context`) for the AI pass. (#43)
24
+
25
+ ## [0.6.0] - 2026-10-15
8
26
 
9
27
  ### Added
10
28
  - `Locallingo::Configuration.parse(text, root_path:, package:)` resolves
@@ -49,7 +67,7 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
49
67
  longer evaluates on `TOPLEVEL_BINDING` (templates still resolve constants
50
68
  from the top level, as the app sees them). (#35)
51
69
 
52
- ## [Unreleased]
70
+ ## [0.5.1] - 2026-10-13
53
71
 
54
72
  ### Added
55
73
  - `Locallingo::Translator#translate(texts, from:, to:, style:, instructions:)`
@@ -73,6 +91,10 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
73
91
  - `require "locallingo/active_record"` adds `Locallingo::Translatable`
74
92
  (`translates_content`), and `Locallingo::RecordTranslator` translates many
75
93
  records in one batched call, writing each under its row lock. (#33)
94
+
95
+ ## [0.5.0] - 2026-10-09
96
+
97
+ ### Added
76
98
  - `lingo quality --prefix yoga.` checks only the keys under a prefix, in both
77
99
  the static checks and the AI pass. A prefix that matches nothing exits `1`. (#10, #29)
78
100
  - `lingo quality --key a.b` (repeatable) checks exactly the named keys, and the
@@ -126,19 +148,46 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
126
148
  and column inside the fence instead of the fence at line 1. (#25)
127
149
  - The translation prompt asks the model to escape `"` inside values as `\"`
128
150
  and keep typographic quotes (“ ” „ ‚ « ») as they appear in the source. (#25)
151
+ - The `duplicate_values` validator reports findings under the configured
152
+ `source_locale` instead of a hard-coded `en`. (#15)
153
+
154
+ ## [0.4.3] - 2026-08-23
155
+
156
+ ### Changed
157
+ - Allow `ruby_llm` 2.x (dependency constraint widened from `< 2` to `< 3`). The
158
+ gem only uses RubyLLM's stable chat API and guarded config accessors, so a
159
+ host app can take the RubyLLM 2.0 bump without waiting on Locallingo.
160
+
161
+ ## [0.4.2] - 2026-08-18
162
+
163
+ ### Fixed
129
164
  - State files are written with a trailing final newline, so the output is
130
165
  POSIX-conformant and files that already end with a newline are no longer
131
166
  rewritten with newline-only diffs. One-time effect after upgrading: each
132
167
  state file gets a single newline-adding rewrite, then stabilizes. (#7)
168
+
169
+ ## [0.4.1] - 2026-07-16
170
+
171
+ ### Fixed
133
172
  - `lingo sync` backfills a missing `target_hash` from the current target value,
134
173
  so hand-added translations (written straight into the YAML, never passing
135
174
  through `translate` or `accept-edits`) get a baseline and the `manual_edits`
136
175
  validator can watch them. An existing `target_hash` is still never
137
176
  recomputed — that would silently absorb hand-edit drift.
138
- - The `duplicate_values` validator reports findings under the configured
139
- `source_locale` instead of a hard-coded `en`. (#15)
140
177
 
141
- ### Fixed (0.4.0)
178
+ ## [0.4.0] - 2026-07-16
179
+
180
+ ### Changed
181
+ - Unscoped `lingo accept-edits` now accepts only the keys the `manual_edits`
182
+ validator flags (actually hand-edited), instead of stamping `manual: true` on
183
+ every key of the locale. Use the new `--key KEY` (repeatable) for surgical
184
+ accepts, or `--all` for the old blanket behavior (initial adoption).
185
+ - Validator suggestions are scoped and safe to follow verbatim: `manual_edit`
186
+ violations suggest `accept-edits --locale <l> --key <key>`, and `outdated`
187
+ violations on manual keys tell you to update the value by hand and re-accept
188
+ it rather than force-translating over curated text.
189
+
190
+ ### Fixed
142
191
  - `lingo sync` no longer destroys hand-edit protection: it now only refreshes
143
192
  each entry's `source_hash` and preserves `target_hash` and `manual: true`
144
193
  (the pre-extraction `bin/translate` behavior). Previously a single sync wiped
@@ -150,15 +199,7 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
150
199
  - State files whose content is unchanged are no longer rewritten, so operations
151
200
  never touch `.i18n-state/` files for unrelated namespaces.
152
201
 
153
- ### Changed
154
- - Unscoped `lingo accept-edits` now accepts only the keys the `manual_edits`
155
- validator flags (actually hand-edited), instead of stamping `manual: true` on
156
- every key of the locale. Use the new `--key KEY` (repeatable) for surgical
157
- accepts, or `--all` for the old blanket behavior (initial adoption).
158
- - Validator suggestions are scoped and safe to follow verbatim: `manual_edit`
159
- violations suggest `accept-edits --locale <l> --key <key>`, and `outdated`
160
- violations on manual keys tell you to update the value by hand and re-accept
161
- it rather than force-translating over curated text.
202
+ ## [0.3.0] - 2026-07-15
162
203
 
163
204
  ### Added
164
205
  - `Locallingo.configure { |c| c.anthropic_api_key = ... }` — gem-level provider
@@ -168,6 +209,21 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
168
209
  - `.locallingo.rb` setup file: the CLI loads it from the project root before
169
210
  dispatch, so standalone `lingo` runs can configure credentials without
170
211
  booting Rails.
212
+
213
+ ## [0.2.2] - 2026-07-13
214
+
215
+ ### Fixed
216
+ - `lingo translate` and `lingo quality --ai` read the provider API key from
217
+ ENV (e.g. `ANTHROPIC_API_KEY`) when RubyLLM has none configured; a key
218
+ configured by the host app still wins. (#2)
219
+
220
+ ## [0.2.1] - 2026-07-07
221
+
222
+ No gem changes (documentation site only).
223
+
224
+ ## [0.2.0] - 2026-07-07
225
+
226
+ ### Added
171
227
  - Initial extraction from the `bin/translate` toolchain into a standalone gem.
172
228
  - `lingo` CLI with subcommands: `status`, `translate`, `validate`, `quality`,
173
229
  `fix-quality`, `accept-edits`, `hash`, `sync`. Legacy `--flag` forms still work
data/README.md CHANGED
@@ -151,6 +151,19 @@ Locallingo::Manager.new(config:).validate # => [violations]
151
151
  Locallingo::QualityChecker.new(config:).check(locale: "en")
152
152
  ```
153
153
 
154
+ `QualityChecker#check_texts(texts, locale:, use_ai: false)` runs the same
155
+ static checks, terminology, British spellings and optional AI review over a
156
+ `{ key => text }` hash instead of the locale files, and returns the same
157
+ suggestion hashes. With `use_ai: true` every given key is reviewed (no
158
+ sampling). `QualityChecker.new(…, context:)` takes a `RubyLLM::Context` for
159
+ the AI pass, as `Translator.new` does:
160
+
161
+ ```ruby
162
+ checker = Locallingo::QualityChecker.new(config:, context: RubyLLM.context { |c| c.anthropic_api_key = key })
163
+ checker.check_texts(result.translations, locale: "de", use_ai: true)
164
+ # => [{ key:, text:, locale:, category:, issue:, match:, severity:, source: :static | :ai }, …]
165
+ ```
166
+
154
167
  `Locallingo::Configuration.parse(yaml, root_path:, package:)` resolves the same
155
168
  config from YAML text (ERB included) instead of reading `.locallingo.yml`. A
156
169
  configuration loaded from the file reads it when it is built, never cached.
@@ -186,9 +199,32 @@ result.failed # => keys that failed after retries
186
199
  - `translate.max_batch_chars` (or `max_batch_chars:`) limits the source
187
200
  characters per batch (a single longer value is sent alone, not split); `translate.pause` (or `pause:`) is the sleep after each
188
201
  batch. Retries of failed or omitted keys keep their own backoff.
189
- - `Locallingo::Prompt.translation(config:, from:, to:, style:, instructions:)`
202
+ - `Locallingo::Prompt.translation(config:, from:, to:, style:, instructions:, placeholders:)`
190
203
  returns the prompt itself.
191
204
 
205
+ Texts with their own token syntax (a template language, an interpolation
206
+ format other than `placeholder_style`) pass `placeholders:`, a list of Regexps
207
+ (default: `translate.placeholders`, regex strings compiled with `Regexp::MULTILINE`, `[]`). The prompt names the
208
+ patterns, and each reply is checked: the tokens every pattern matches in a
209
+ value must equal its source's, compared on the exact text and by count (order
210
+ may change). A value that differs is re-requested in the omitted-key rounds;
211
+ one that still differs is never returned, but listed in `failed` and in
212
+ `rejections`:
213
+
214
+ ```ruby
215
+ LIQUID = [/\{\{.*?\}\}/m, /\{%.*?%\}/m]
216
+
217
+ result = translator.translate(texts, from: "en", to: "de", style: :prose, placeholders: LIQUID)
218
+ result.translations # only values whose tokens match their source
219
+ result.failed # => ["body"]
220
+ result.rejections # => { "body" => { "missing" => ["{% paragraph %}"], "extra" => [] } }
221
+ ```
222
+
223
+ `Locallingo::Placeholders.compare(source, translated, patterns)` is the check
224
+ itself (`nil` when the tokens match). `RecordTranslator#call` takes
225
+ `placeholders:` too; a record with a rejected segment is not written. The
226
+ `lingo` CLI does not use `translate.placeholders`.
227
+
192
228
  `Locallingo::Drift` is the missing / outdated / hand-edited rule over the state
193
229
  entry shape (`source_hash`, `target_hash`, `manual`), for strings or nested
194
230
  values:
@@ -32,6 +32,11 @@ defaults:
32
32
  max_batch_chars: null
33
33
  # Seconds to sleep after each batch (rate-limit courtesy). 0 disables it.
34
34
  pause: 1.0
35
+ # Regex patterns (strings) for tokens the Translator API must keep exactly,
36
+ # e.g. ['\{\{.*?\}\}', '\{%.*?%\}'] for Liquid (`.` spans lines). A reply whose tokens differ
37
+ # from its source (by count) is retried, then reported as failed. The
38
+ # `lingo` CLI does not use it. [] = no check.
39
+ placeholders: []
35
40
 
36
41
  quality:
37
42
  model: gpt-4o-mini
@@ -11,6 +11,8 @@ module Locallingo
11
11
  # down to single keys, so a bad value costs only its own key.
12
12
  # - Keys the model omitted are re-batched for a few rounds; keys whose own
13
13
  # one-key reply could not be parsed are not.
14
+ # - A value the `accept` callable rejects is treated like an omitted key: it
15
+ # joins those rounds, and is reported with its reason if still rejected.
14
16
  class BatchTranslator
15
17
  MAX_RETRIES = 3
16
18
  MAX_MISSING_RETRIES = 2
@@ -21,28 +23,34 @@ module Locallingo
21
23
  # A batch closes at `batch_size` keys or `max_batch_chars` characters of
22
24
  # source text, whichever comes first; a single oversized value goes alone.
23
25
  # `pause` is the sleep after each batch (0 in a background job).
24
- def initialize(provider:, model:, batch_size:, log:, max_batch_chars: nil, pause: BASE_SLEEP_DURATION)
26
+ # `accept` is called as `accept.call(key, source_text, translated_text)`
27
+ # and returns true to keep the value, or a Hash: the rejection reason.
28
+ def initialize(provider:, model:, batch_size:, log:, max_batch_chars: nil, pause: BASE_SLEEP_DURATION,
29
+ accept: nil)
25
30
  @provider = provider
26
31
  @model = model
27
32
  @batch_size = batch_size
28
33
  @max_batch_chars = max_batch_chars
29
34
  @pause = pause
30
35
  @log = log
36
+ @accept = accept
31
37
  end
32
38
 
33
39
  # Translates `keys` (looked up in `source`) and returns
34
- # `[translations, failed_keys]`.
40
+ # `[translations, failed_keys, rejections]`; `rejections` maps each failed
41
+ # key whose reply in its last round `accept` rejected to the reason.
35
42
  def call(source, keys, instructions:)
36
43
  @source = source
37
44
  @instructions = instructions
38
45
  @unparseable = []
46
+ @rejections = {}
39
47
 
40
48
  translated, missing = translate_keys(keys)
41
49
  missing = retry_missing(translated, missing)
42
50
 
43
51
  failed = missing + @unparseable
44
52
  report_failed(failed)
45
- [translated, failed]
53
+ [translated, failed, @rejections]
46
54
  end
47
55
 
48
56
  private
@@ -75,7 +83,8 @@ module Locallingo
75
83
  batches(keys).each do |batch|
76
84
  result = translate_batch(batch.to_h { |key| [key, @source[key]] })
77
85
  batch.each do |key|
78
- if result.key?(key) && !result[key].to_s.empty?
86
+ @rejections.delete(key)
87
+ if result.key?(key) && !result[key].to_s.empty? && accepted?(key, result[key])
79
88
  translations[key] = result[key]
80
89
  else
81
90
  failed << key
@@ -87,6 +96,17 @@ module Locallingo
87
96
  [translations, failed]
88
97
  end
89
98
 
99
+ def accepted?(key, value)
100
+ return true unless @accept
101
+
102
+ verdict = @accept.call(key, @source[key], value)
103
+ return true if verdict == true
104
+
105
+ @rejections[key] = verdict.is_a?(Hash) ? verdict : {}
106
+ log(" Rejected #{key}: #{@rejections[key]}", level: :warn)
107
+ false
108
+ end
109
+
90
110
  def batches(keys)
91
111
  return keys.each_slice(@batch_size).to_a unless @max_batch_chars
92
112
 
@@ -72,6 +72,13 @@ module Locallingo
72
72
  def british_spellings? = !!dig("quality", "british_spellings")
73
73
  def terminology_setting = dig("quality", "terminology")
74
74
 
75
+ # Token patterns (`translate.placeholders`, regex strings compiled with
76
+ # Regexp::MULTILINE, so `.` spans lines) whose matches a Translator must
77
+ # keep unchanged; [] by default.
78
+ def placeholders
79
+ Array(dig("translate", "placeholders")).map { |pattern| compile_placeholder(pattern) }.freeze
80
+ end
81
+
75
82
  # --- prompt scaffolding --------------------------------------------------
76
83
 
77
84
  def context = data.fetch("context", "a business application")
@@ -123,6 +130,17 @@ module Locallingo
123
130
 
124
131
  def dig(*keys) = data.dig(*keys)
125
132
 
133
+ def compile_placeholder(pattern)
134
+ return pattern.dup.freeze if pattern.is_a?(Regexp)
135
+ unless pattern.is_a?(String)
136
+ raise Error, "translate.placeholders: expected a regex string, got #{pattern.inspect}"
137
+ end
138
+
139
+ Regexp.new(pattern, Regexp::MULTILINE).freeze
140
+ rescue RegexpError => e
141
+ raise Error, "translate.placeholders: #{pattern.to_s.inspect} is not a valid regular expression (#{e.message})"
142
+ end
143
+
126
144
  # Merge: shipped defaults <- user `defaults:` <- selected `packages:` entry.
127
145
  def resolve(source)
128
146
  document, origin = user_config(source)
@@ -0,0 +1,35 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Locallingo
4
+ # Token checks for translated text: the tokens a list of patterns matches
5
+ # (`{{ item.name }}`, `%{count}`, …) must survive translation unchanged, in
6
+ # the same count. Order may change, since word order does. Pure functions
7
+ # over their arguments.
8
+ module Placeholders
9
+ module_function
10
+
11
+ # Every whole match of every pattern in +text+, pattern by pattern.
12
+ def tokens(text, patterns)
13
+ patterns.flat_map { |pattern| text.to_s.gsub(pattern).to_a }
14
+ end
15
+
16
+ # nil when +translated+ holds exactly the tokens of +source+, else
17
+ # `{ "missing" => [...], "extra" => [...] }` (multiset difference on the
18
+ # exact matched text).
19
+ def compare(source, translated, patterns)
20
+ before = tokens(source, patterns).tally
21
+ after = tokens(translated, patterns).tally
22
+ missing = difference(before, after)
23
+ extra = difference(after, before)
24
+ return nil if missing.empty? && extra.empty?
25
+
26
+ { "missing" => missing, "extra" => extra }
27
+ end
28
+
29
+ def difference(counts, other)
30
+ counts.flat_map { |token, count| [token] * [count - other.fetch(token, 0), 0].max }
31
+ end
32
+
33
+ private_class_method :difference
34
+ end
35
+ end
@@ -22,7 +22,7 @@ module Locallingo
22
22
 
23
23
  module_function
24
24
 
25
- def translation(config:, from:, to:, style: :ui, instructions: nil)
25
+ def translation(config:, from:, to:, style: :ui, instructions: nil, placeholders: [])
26
26
  raise Error, "Unknown prompt style #{style.inspect} (expected one of #{STYLES.join(", ")})" unless
27
27
  STYLES.include?(style.to_s.to_sym)
28
28
 
@@ -41,7 +41,7 @@ module Locallingo
41
41
 
42
42
  ## General Rules
43
43
  - Preserve placeholders like #{config.placeholder_style} exactly as they appear
44
- - Preserve HTML tags if present
44
+ #{placeholders_rule(placeholders)}- Preserve HTML tags if present
45
45
  #{style.to_sym == :prose ? MARKDOWN_RULES : BREVITY_RULE}- Escape every double quote (") inside a translated value as \\" so the reply stays valid JSON
46
46
  - Keep typographic quotes (“ ” „ ‚ « ») exactly as they appear in the source
47
47
  #{glossary_section(config)}#{language_guide_section(config, to)}#{instructions_section(instructions)}
@@ -51,6 +51,14 @@ module Locallingo
51
51
  PROMPT
52
52
  end
53
53
 
54
+ def placeholders_rule(patterns)
55
+ patterns = Array(patterns)
56
+ return "" if patterns.empty?
57
+
58
+ sources = patterns.map { |pattern| "`#{pattern.source}`" }.join(", ")
59
+ "- Preserve every token matching these patterns exactly as it appears, with the same count: #{sources}\n"
60
+ end
61
+
54
62
  def glossary_section(config)
55
63
  return "" if config.glossary.empty?
56
64
 
@@ -68,6 +76,6 @@ module Locallingo
68
76
  text.empty? ? "" : "\n## Additional Instructions\n#{text}\n"
69
77
  end
70
78
 
71
- private_class_method :glossary_section, :language_guide_section, :instructions_section
79
+ private_class_method :placeholders_rule, :glossary_section, :language_guide_section, :instructions_section
72
80
  end
73
81
  end
@@ -16,6 +16,9 @@ module Locallingo
16
16
  # optional British-spelling drift, and a very-long-text heuristic. An optional
17
17
  # AI pass (RubyLLM) reviews a sample for clarity/professionalism.
18
18
  #
19
+ # `check` and `check_key` read the locale files; `check_texts` runs the same
20
+ # checks over a `{ key => text }` hash held anywhere else.
21
+ #
19
22
  # `fix!` rewrites the auto-fixable suggestions (universal fixes + British
20
23
  # spellings) back into the locale files, preserving case.
21
24
  class QualityChecker
@@ -23,11 +26,13 @@ module Locallingo
23
26
 
24
27
  attr_reader :config, :verbose, :logger
25
28
 
26
- def initialize(config: nil, root_path: nil, package: nil, verbose: false, logger: nil)
29
+ # +context+ is an optional RubyLLM::Context the AI pass runs on (e.g. a
30
+ # per-call key), as for Translator.
31
+ def initialize(config: nil, root_path: nil, package: nil, verbose: false, logger: nil, context: nil)
27
32
  @config = config || Configuration.load(root_path: root_path || Dir.pwd, package:)
28
33
  @verbose = verbose
29
34
  @logger = logger
30
- @provider = Providers::RubyLLM.new(provider: @config.provider)
35
+ @provider = Providers::RubyLLM.new(provider: @config.provider, context:)
31
36
  @terminology = Quality::Terminology.new(@config.terminology_setting, base_path: @config.base_path)
32
37
  end
33
38
 
@@ -37,20 +42,26 @@ module Locallingo
37
42
  def check(locale: nil, use_ai: false, prefix: nil, keys: nil)
38
43
  locale ||= config.source_locale
39
44
  translations = scoped_translations(locale, prefix:, keys:)
45
+ return check_texts(translations, locale:, use_ai:) if keys || !use_ai
40
46
 
41
- suggestions = translations.flat_map { |key, text| check_text(key, text, locale) }
42
- suggestions.concat(keys ? suggest_improvements(translations, locale) : ai_sample(translations, locale)) if use_ai
43
- suggestions
47
+ check_texts(translations, locale:).concat(ai_sample(translations, locale))
44
48
  end
45
49
 
46
50
  # Check a single key.
47
51
  def check_key(key, locale, use_ai: false)
48
- translations = load_locale_translations(locale)
49
- text = translations[key]
52
+ text = load_locale_translations(locale)[key]
50
53
  return [{ key:, error: "Key not found" }] unless text
51
54
 
52
- suggestions = check_text(key, text, locale)
53
- suggestions.concat(suggest_improvements({ key => text }, locale)) if use_ai
55
+ check_texts({ key => text }, locale:, use_ai:)
56
+ end
57
+
58
+ # Check a `{ key => text }` hash without reading any file: the static
59
+ # checks on every pair, then, with +use_ai+, the AI pass over every given
60
+ # key (unsampled). Returns the suggestion hashes `check` returns.
61
+ def check_texts(texts, locale:, use_ai: false)
62
+ texts = stringify_keys(texts).select { |_key, text| text.is_a?(String) }
63
+ suggestions = texts.flat_map { |key, text| check_text(key, text, locale) }
64
+ suggestions.concat(suggest_improvements(texts, locale)) if use_ai && texts.any?
54
65
  suggestions
55
66
  end
56
67
 
@@ -105,6 +116,14 @@ module Locallingo
105
116
  suggestions.compact
106
117
  end
107
118
 
119
+ def stringify_keys(texts)
120
+ texts.each_with_object({}) do |(key, text), all|
121
+ raise Error, "Duplicate key #{key.to_s.inspect} (e.g. :#{key} and \"#{key}\")" if all.key?(key.to_s)
122
+
123
+ all[key.to_s] = text
124
+ end
125
+ end
126
+
108
127
  def british_spellings_for?(locale)
109
128
  config.british_spellings? && locale.to_s == config.source_locale.to_s
110
129
  end
@@ -13,7 +13,8 @@ module Locallingo
13
13
  # its row lock, after re-checking that its source is unchanged and the
14
14
  # target is still due: a source edited or a target hand-written while the
15
15
  # model answered is skipped and left for the next run. A record with any
16
- # failed segment is not written; it stays due and its paths are reported.
16
+ # failed segment (including one whose tokens did not survive, with
17
+ # `placeholders:`) is not written; it stays due and its paths are reported.
17
18
  #
18
19
  # Records must be persisted and of one model (ids are the key prefix). This
19
20
  # class never loads ActiveRecord itself; records only need the concern.
@@ -30,7 +31,8 @@ module Locallingo
30
31
  @write = write
31
32
  end
32
33
 
33
- def call(records, to:, style: :ui, instructions: nil, force: false, include_manual: false)
34
+ # +placeholders+ is passed to Translator#translate (nil: its default).
35
+ def call(records, to:, style: :ui, instructions: nil, force: false, include_manual: false, placeholders: nil)
34
36
  @locale = to.to_s
35
37
  @force = force
36
38
  @include_manual = include_manual
@@ -39,7 +41,7 @@ module Locallingo
39
41
  due, empty = records.select { |record| due?(record) }.partition { |record| record.translation_segments.any? }
40
42
  @result[:skipped].concat(empty.map(&:id))
41
43
  due.group_by(&:translation_source_locale).each do |from, group|
42
- translate_group(group, from:, style:, instructions:)
44
+ translate_group(group, from:, style:, instructions:, placeholders:)
43
45
  end
44
46
  @result
45
47
  end
@@ -48,7 +50,7 @@ module Locallingo
48
50
 
49
51
  def due?(record) = record.translation_due?(@locale, force: @force, include_manual: @include_manual)
50
52
 
51
- def translate_group(records, from:, style:, instructions:)
53
+ def translate_group(records, from:, style:, instructions:, placeholders:)
52
54
  digests = records.to_h { |record| [record, record.translation_source_digest] }
53
55
  owners = {}
54
56
  segments = records.each_with_object({}) do |record, all|
@@ -61,7 +63,7 @@ module Locallingo
61
63
  end
62
64
  end
63
65
 
64
- result = @translator.translate(segments, from:, to: @locale, style:, instructions:)
66
+ result = @translator.translate(segments, from:, to: @locale, style:, instructions:, placeholders:)
65
67
  records.each do |record|
66
68
  translations = own(result.translations, owners, record).to_h
67
69
  failed = own(result.failed.to_h { |key| [key, nil] }, owners, record).map(&:first)
@@ -6,6 +6,7 @@ require_relative "configuration"
6
6
  require_relative "prompt"
7
7
  require_relative "providers/ruby_llm"
8
8
  require_relative "batch_translator"
9
+ require_relative "placeholders"
9
10
 
10
11
  module Locallingo
11
12
  # The public, file-agnostic entry point: translates a plain `{ key => text }`
@@ -19,11 +20,19 @@ module Locallingo
19
20
  # result = translator.translate({ "title" => "Hello" }, from: "en", to: "de", style: :prose)
20
21
  # result.translations # => { "title" => "Hallo" }
21
22
  # result.failed # => []
23
+ # result.rejections # => {}
22
24
  #
23
25
  # Batching, retries, batch halving and omitted-key rounds come from
24
26
  # BatchTranslator. Blank texts are not sent.
27
+ #
28
+ # With `placeholders:` (Regexps; default `translate.placeholders`), a value
29
+ # whose tokens differ from its source's is re-requested in those rounds and,
30
+ # if it still differs, reported in `failed` and `rejections` instead of
31
+ # returned.
25
32
  class Translator
26
- Result = Data.define(:translations, :failed)
33
+ Result = Data.define(:translations, :failed, :rejections) do
34
+ def initialize(translations:, failed:, rejections: {}) = super
35
+ end
27
36
 
28
37
  attr_reader :config
29
38
 
@@ -38,15 +47,16 @@ module Locallingo
38
47
  end
39
48
 
40
49
  # Raises MissingCredentialsError before any call when no key is available.
41
- def translate(texts, from:, to:, style: :ui, instructions: nil)
50
+ def translate(texts, from:, to:, style: :ui, instructions: nil, placeholders: nil)
51
+ patterns = placeholder_patterns(placeholders)
42
52
  @provider.ensure_credentials!
43
53
 
44
54
  source = stringify_keys(texts).reject { |_key, text| text.to_s.strip.empty? }
45
- return Result.new(translations: {}, failed: []) if source.empty?
55
+ return Result.new(translations: {}, failed: [], rejections: {}) if source.empty?
46
56
 
47
- prompt = Prompt.translation(config:, from:, to:, style:, instructions:)
48
- translated, failed = batch_translator.call(source, source.keys, instructions: prompt)
49
- Result.new(translations: translated.slice(*source.keys).except(*failed), failed:)
57
+ prompt = Prompt.translation(config:, from:, to:, style:, instructions:, placeholders: patterns)
58
+ translated, failed, rejections = batch_translator(patterns).call(source, source.keys, instructions: prompt)
59
+ Result.new(translations: translated.slice(*source.keys).except(*failed), failed:, rejections:)
50
60
  end
51
61
 
52
62
  private
@@ -59,12 +69,28 @@ module Locallingo
59
69
  end
60
70
  end
61
71
 
62
- def batch_translator
63
- @batch_translator ||= BatchTranslator.new(
72
+ def placeholder_patterns(placeholders)
73
+ patterns = placeholders.nil? ? config.placeholders : Array(placeholders)
74
+ invalid = patterns.grep_v(Regexp)
75
+ raise Error, "placeholders: expected Regexps, got #{invalid.map(&:inspect).join(", ")}" if invalid.any?
76
+
77
+ patterns
78
+ end
79
+
80
+ # One per call: `accept` closes over that call's patterns.
81
+ def batch_translator(patterns)
82
+ BatchTranslator.new(
64
83
  provider: @provider, model: @model, batch_size: config.batch_size,
65
84
  max_batch_chars: @max_batch_chars, pause: @pause,
66
- log: ->(message, level: :info) { @logger.public_send(level, message) }
85
+ log: ->(message, level: :info) { @logger.public_send(level, message) },
86
+ accept: accept_for(patterns)
67
87
  )
68
88
  end
89
+
90
+ def accept_for(patterns)
91
+ return nil if patterns.empty?
92
+
93
+ ->(_key, source_text, translated) { Placeholders.compare(source_text, translated, patterns) || true }
94
+ end
69
95
  end
70
96
  end
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module Locallingo
4
- VERSION = "0.6.0"
4
+ VERSION = "0.6.1"
5
5
  end
data/lib/locallingo.rb CHANGED
@@ -9,6 +9,7 @@ require_relative "locallingo/key_flattener"
9
9
  require_relative "locallingo/state_store"
10
10
  require_relative "locallingo/drift"
11
11
  require_relative "locallingo/providers/ruby_llm"
12
+ require_relative "locallingo/placeholders"
12
13
  require_relative "locallingo/batch_translator"
13
14
  require_relative "locallingo/prompt"
14
15
  require_relative "locallingo/translator"
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: locallingo
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.6.0
4
+ version: 0.6.1
5
5
  platform: ruby
6
6
  authors:
7
7
  - Mikael Henriksson
@@ -78,6 +78,7 @@ files:
78
78
  - lib/locallingo/json_extraction.rb
79
79
  - lib/locallingo/key_flattener.rb
80
80
  - lib/locallingo/manager.rb
81
+ - lib/locallingo/placeholders.rb
81
82
  - lib/locallingo/prompt.rb
82
83
  - lib/locallingo/providers/ruby_llm.rb
83
84
  - lib/locallingo/quality/british_spellings.rb