locallingo 0.6.0 → 0.6.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +70 -14
- data/README.md +37 -1
- data/config/locallingo.default.yml +5 -0
- data/lib/locallingo/batch_translator.rb +24 -4
- data/lib/locallingo/configuration.rb +18 -0
- data/lib/locallingo/placeholders.rb +35 -0
- data/lib/locallingo/prompt.rb +11 -3
- data/lib/locallingo/quality_checker.rb +28 -9
- data/lib/locallingo/record_translator.rb +7 -5
- data/lib/locallingo/translator.rb +35 -9
- data/lib/locallingo/version.rb +1 -1
- data/lib/locallingo.rb +1 -0
- metadata +2 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: '08b89ac361ed67239fe3e8574396cf5ffb1b0fb20a04385cea02d350b608d503'
|
|
4
|
+
data.tar.gz: 2de02e7a1ca34d339bff982427a411bb299c02907d3d8545e8dcb1092a747031
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: b488c4209a18715d8772e2616b2bf80eeafe0973d0a97be9fc5da8b01417308550aac86fc3b512fec3cd480964aac0281e02a126319a882aaf4fb49776748bae
|
|
7
|
+
data.tar.gz: b4c13e6f0ca99cb0a0a54b1d56be91dad3b5810cf40f771ac653e63ec2f2ab3bf24fa460e1ecb661c7ac280f0ef4961df1a9b97070a991f8fce6c39244b14666
|
data/CHANGELOG.md
CHANGED
|
@@ -4,7 +4,25 @@ All notable changes to this project are documented here. The format is based on
|
|
|
4
4
|
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project
|
|
5
5
|
adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
6
6
|
|
|
7
|
-
## [
|
|
7
|
+
## [Unreleased]
|
|
8
|
+
|
|
9
|
+
### Added
|
|
10
|
+
- `Translator#translate(…, placeholders:)` takes a list of Regexps (default:
|
|
11
|
+
the new `translate.placeholders` config key, regex strings, `[]`) for the
|
|
12
|
+
tokens the texts use. The prompt names the patterns, and a value whose
|
|
13
|
+
tokens differ from its source's (exact text, by count) is re-requested in
|
|
14
|
+
the omitted-key rounds, then reported in `failed` instead of returned.
|
|
15
|
+
`Locallingo::Placeholders` is the check. (#43)
|
|
16
|
+
- `Translator::Result#rejections` (`{ key => { "missing" => [...], "extra" =>
|
|
17
|
+
[...] } }`) and `BatchTranslator.new(…, accept:)`, whose `call` now returns
|
|
18
|
+
`[translations, failed, rejections]`. `RecordTranslator#call` passes
|
|
19
|
+
`placeholders:` through. The `lingo` CLI is unchanged. (#43)
|
|
20
|
+
- `QualityChecker#check_texts(texts, locale:, use_ai: false)` runs the static
|
|
21
|
+
checks, terminology, British spellings and the optional AI review over a
|
|
22
|
+
`{ key => text }` hash instead of the locale files; `QualityChecker.new`
|
|
23
|
+
takes `context:` (a `RubyLLM::Context`) for the AI pass. (#43)
|
|
24
|
+
|
|
25
|
+
## [0.6.0] - 2026-10-15
|
|
8
26
|
|
|
9
27
|
### Added
|
|
10
28
|
- `Locallingo::Configuration.parse(text, root_path:, package:)` resolves
|
|
@@ -49,7 +67,7 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
|
49
67
|
longer evaluates on `TOPLEVEL_BINDING` (templates still resolve constants
|
|
50
68
|
from the top level, as the app sees them). (#35)
|
|
51
69
|
|
|
52
|
-
## [
|
|
70
|
+
## [0.5.1] - 2026-10-13
|
|
53
71
|
|
|
54
72
|
### Added
|
|
55
73
|
- `Locallingo::Translator#translate(texts, from:, to:, style:, instructions:)`
|
|
@@ -73,6 +91,10 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
|
73
91
|
- `require "locallingo/active_record"` adds `Locallingo::Translatable`
|
|
74
92
|
(`translates_content`), and `Locallingo::RecordTranslator` translates many
|
|
75
93
|
records in one batched call, writing each under its row lock. (#33)
|
|
94
|
+
|
|
95
|
+
## [0.5.0] - 2026-10-09
|
|
96
|
+
|
|
97
|
+
### Added
|
|
76
98
|
- `lingo quality --prefix yoga.` checks only the keys under a prefix, in both
|
|
77
99
|
the static checks and the AI pass. A prefix that matches nothing exits `1`. (#10, #29)
|
|
78
100
|
- `lingo quality --key a.b` (repeatable) checks exactly the named keys, and the
|
|
@@ -126,19 +148,46 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
|
126
148
|
and column inside the fence instead of the fence at line 1. (#25)
|
|
127
149
|
- The translation prompt asks the model to escape `"` inside values as `\"`
|
|
128
150
|
and keep typographic quotes (“ ” „ ‚ « ») as they appear in the source. (#25)
|
|
151
|
+
- The `duplicate_values` validator reports findings under the configured
|
|
152
|
+
`source_locale` instead of a hard-coded `en`. (#15)
|
|
153
|
+
|
|
154
|
+
## [0.4.3] - 2026-08-23
|
|
155
|
+
|
|
156
|
+
### Changed
|
|
157
|
+
- Allow `ruby_llm` 2.x (dependency constraint widened from `< 2` to `< 3`). The
|
|
158
|
+
gem only uses RubyLLM's stable chat API and guarded config accessors, so a
|
|
159
|
+
host app can take the RubyLLM 2.0 bump without waiting on Locallingo.
|
|
160
|
+
|
|
161
|
+
## [0.4.2] - 2026-08-18
|
|
162
|
+
|
|
163
|
+
### Fixed
|
|
129
164
|
- State files are written with a trailing final newline, so the output is
|
|
130
165
|
POSIX-conformant and files that already end with a newline are no longer
|
|
131
166
|
rewritten with newline-only diffs. One-time effect after upgrading: each
|
|
132
167
|
state file gets a single newline-adding rewrite, then stabilizes. (#7)
|
|
168
|
+
|
|
169
|
+
## [0.4.1] - 2026-07-16
|
|
170
|
+
|
|
171
|
+
### Fixed
|
|
133
172
|
- `lingo sync` backfills a missing `target_hash` from the current target value,
|
|
134
173
|
so hand-added translations (written straight into the YAML, never passing
|
|
135
174
|
through `translate` or `accept-edits`) get a baseline and the `manual_edits`
|
|
136
175
|
validator can watch them. An existing `target_hash` is still never
|
|
137
176
|
recomputed — that would silently absorb hand-edit drift.
|
|
138
|
-
- The `duplicate_values` validator reports findings under the configured
|
|
139
|
-
`source_locale` instead of a hard-coded `en`. (#15)
|
|
140
177
|
|
|
141
|
-
|
|
178
|
+
## [0.4.0] - 2026-07-16
|
|
179
|
+
|
|
180
|
+
### Changed
|
|
181
|
+
- Unscoped `lingo accept-edits` now accepts only the keys the `manual_edits`
|
|
182
|
+
validator flags (actually hand-edited), instead of stamping `manual: true` on
|
|
183
|
+
every key of the locale. Use the new `--key KEY` (repeatable) for surgical
|
|
184
|
+
accepts, or `--all` for the old blanket behavior (initial adoption).
|
|
185
|
+
- Validator suggestions are scoped and safe to follow verbatim: `manual_edit`
|
|
186
|
+
violations suggest `accept-edits --locale <l> --key <key>`, and `outdated`
|
|
187
|
+
violations on manual keys tell you to update the value by hand and re-accept
|
|
188
|
+
it rather than force-translating over curated text.
|
|
189
|
+
|
|
190
|
+
### Fixed
|
|
142
191
|
- `lingo sync` no longer destroys hand-edit protection: it now only refreshes
|
|
143
192
|
each entry's `source_hash` and preserves `target_hash` and `manual: true`
|
|
144
193
|
(the pre-extraction `bin/translate` behavior). Previously a single sync wiped
|
|
@@ -150,15 +199,7 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
|
150
199
|
- State files whose content is unchanged are no longer rewritten, so operations
|
|
151
200
|
never touch `.i18n-state/` files for unrelated namespaces.
|
|
152
201
|
|
|
153
|
-
|
|
154
|
-
- Unscoped `lingo accept-edits` now accepts only the keys the `manual_edits`
|
|
155
|
-
validator flags (actually hand-edited), instead of stamping `manual: true` on
|
|
156
|
-
every key of the locale. Use the new `--key KEY` (repeatable) for surgical
|
|
157
|
-
accepts, or `--all` for the old blanket behavior (initial adoption).
|
|
158
|
-
- Validator suggestions are scoped and safe to follow verbatim: `manual_edit`
|
|
159
|
-
violations suggest `accept-edits --locale <l> --key <key>`, and `outdated`
|
|
160
|
-
violations on manual keys tell you to update the value by hand and re-accept
|
|
161
|
-
it rather than force-translating over curated text.
|
|
202
|
+
## [0.3.0] - 2026-07-15
|
|
162
203
|
|
|
163
204
|
### Added
|
|
164
205
|
- `Locallingo.configure { |c| c.anthropic_api_key = ... }` — gem-level provider
|
|
@@ -168,6 +209,21 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
|
168
209
|
- `.locallingo.rb` setup file: the CLI loads it from the project root before
|
|
169
210
|
dispatch, so standalone `lingo` runs can configure credentials without
|
|
170
211
|
booting Rails.
|
|
212
|
+
|
|
213
|
+
## [0.2.2] - 2026-07-13
|
|
214
|
+
|
|
215
|
+
### Fixed
|
|
216
|
+
- `lingo translate` and `lingo quality --ai` read the provider API key from
|
|
217
|
+
ENV (e.g. `ANTHROPIC_API_KEY`) when RubyLLM has none configured; a key
|
|
218
|
+
configured by the host app still wins. (#2)
|
|
219
|
+
|
|
220
|
+
## [0.2.1] - 2026-07-07
|
|
221
|
+
|
|
222
|
+
No gem changes (documentation site only).
|
|
223
|
+
|
|
224
|
+
## [0.2.0] - 2026-07-07
|
|
225
|
+
|
|
226
|
+
### Added
|
|
171
227
|
- Initial extraction from the `bin/translate` toolchain into a standalone gem.
|
|
172
228
|
- `lingo` CLI with subcommands: `status`, `translate`, `validate`, `quality`,
|
|
173
229
|
`fix-quality`, `accept-edits`, `hash`, `sync`. Legacy `--flag` forms still work
|
data/README.md
CHANGED
|
@@ -151,6 +151,19 @@ Locallingo::Manager.new(config:).validate # => [violations]
|
|
|
151
151
|
Locallingo::QualityChecker.new(config:).check(locale: "en")
|
|
152
152
|
```
|
|
153
153
|
|
|
154
|
+
`QualityChecker#check_texts(texts, locale:, use_ai: false)` runs the same
|
|
155
|
+
static checks, terminology, British spellings and optional AI review over a
|
|
156
|
+
`{ key => text }` hash instead of the locale files, and returns the same
|
|
157
|
+
suggestion hashes. With `use_ai: true` every given key is reviewed (no
|
|
158
|
+
sampling). `QualityChecker.new(…, context:)` takes a `RubyLLM::Context` for
|
|
159
|
+
the AI pass, as `Translator.new` does:
|
|
160
|
+
|
|
161
|
+
```ruby
|
|
162
|
+
checker = Locallingo::QualityChecker.new(config:, context: RubyLLM.context { |c| c.anthropic_api_key = key })
|
|
163
|
+
checker.check_texts(result.translations, locale: "de", use_ai: true)
|
|
164
|
+
# => [{ key:, text:, locale:, category:, issue:, match:, severity:, source: :static | :ai }, …]
|
|
165
|
+
```
|
|
166
|
+
|
|
154
167
|
`Locallingo::Configuration.parse(yaml, root_path:, package:)` resolves the same
|
|
155
168
|
config from YAML text (ERB included) instead of reading `.locallingo.yml`. A
|
|
156
169
|
configuration loaded from the file reads it when it is built, never cached.
|
|
@@ -186,9 +199,32 @@ result.failed # => keys that failed after retries
|
|
|
186
199
|
- `translate.max_batch_chars` (or `max_batch_chars:`) limits the source
|
|
187
200
|
characters per batch (a single longer value is sent alone, not split); `translate.pause` (or `pause:`) is the sleep after each
|
|
188
201
|
batch. Retries of failed or omitted keys keep their own backoff.
|
|
189
|
-
- `Locallingo::Prompt.translation(config:, from:, to:, style:, instructions:)`
|
|
202
|
+
- `Locallingo::Prompt.translation(config:, from:, to:, style:, instructions:, placeholders:)`
|
|
190
203
|
returns the prompt itself.
|
|
191
204
|
|
|
205
|
+
Texts with their own token syntax (a template language, an interpolation
|
|
206
|
+
format other than `placeholder_style`) pass `placeholders:`, a list of Regexps
|
|
207
|
+
(default: `translate.placeholders`, regex strings compiled with `Regexp::MULTILINE`, `[]`). The prompt names the
|
|
208
|
+
patterns, and each reply is checked: the tokens every pattern matches in a
|
|
209
|
+
value must equal its source's, compared on the exact text and by count (order
|
|
210
|
+
may change). A value that differs is re-requested in the omitted-key rounds;
|
|
211
|
+
one that still differs is never returned, but listed in `failed` and in
|
|
212
|
+
`rejections`:
|
|
213
|
+
|
|
214
|
+
```ruby
|
|
215
|
+
LIQUID = [/\{\{.*?\}\}/m, /\{%.*?%\}/m]
|
|
216
|
+
|
|
217
|
+
result = translator.translate(texts, from: "en", to: "de", style: :prose, placeholders: LIQUID)
|
|
218
|
+
result.translations # only values whose tokens match their source
|
|
219
|
+
result.failed # => ["body"]
|
|
220
|
+
result.rejections # => { "body" => { "missing" => ["{% paragraph %}"], "extra" => [] } }
|
|
221
|
+
```
|
|
222
|
+
|
|
223
|
+
`Locallingo::Placeholders.compare(source, translated, patterns)` is the check
|
|
224
|
+
itself (`nil` when the tokens match). `RecordTranslator#call` takes
|
|
225
|
+
`placeholders:` too; a record with a rejected segment is not written. The
|
|
226
|
+
`lingo` CLI does not use `translate.placeholders`.
|
|
227
|
+
|
|
192
228
|
`Locallingo::Drift` is the missing / outdated / hand-edited rule over the state
|
|
193
229
|
entry shape (`source_hash`, `target_hash`, `manual`), for strings or nested
|
|
194
230
|
values:
|
|
@@ -32,6 +32,11 @@ defaults:
|
|
|
32
32
|
max_batch_chars: null
|
|
33
33
|
# Seconds to sleep after each batch (rate-limit courtesy). 0 disables it.
|
|
34
34
|
pause: 1.0
|
|
35
|
+
# Regex patterns (strings) for tokens the Translator API must keep exactly,
|
|
36
|
+
# e.g. ['\{\{.*?\}\}', '\{%.*?%\}'] for Liquid (`.` spans lines). A reply whose tokens differ
|
|
37
|
+
# from its source (by count) is retried, then reported as failed. The
|
|
38
|
+
# `lingo` CLI does not use it. [] = no check.
|
|
39
|
+
placeholders: []
|
|
35
40
|
|
|
36
41
|
quality:
|
|
37
42
|
model: gpt-4o-mini
|
|
@@ -11,6 +11,8 @@ module Locallingo
|
|
|
11
11
|
# down to single keys, so a bad value costs only its own key.
|
|
12
12
|
# - Keys the model omitted are re-batched for a few rounds; keys whose own
|
|
13
13
|
# one-key reply could not be parsed are not.
|
|
14
|
+
# - A value the `accept` callable rejects is treated like an omitted key: it
|
|
15
|
+
# joins those rounds, and is reported with its reason if still rejected.
|
|
14
16
|
class BatchTranslator
|
|
15
17
|
MAX_RETRIES = 3
|
|
16
18
|
MAX_MISSING_RETRIES = 2
|
|
@@ -21,28 +23,34 @@ module Locallingo
|
|
|
21
23
|
# A batch closes at `batch_size` keys or `max_batch_chars` characters of
|
|
22
24
|
# source text, whichever comes first; a single oversized value goes alone.
|
|
23
25
|
# `pause` is the sleep after each batch (0 in a background job).
|
|
24
|
-
|
|
26
|
+
# `accept` is called as `accept.call(key, source_text, translated_text)`
|
|
27
|
+
# and returns true to keep the value, or a Hash: the rejection reason.
|
|
28
|
+
def initialize(provider:, model:, batch_size:, log:, max_batch_chars: nil, pause: BASE_SLEEP_DURATION,
|
|
29
|
+
accept: nil)
|
|
25
30
|
@provider = provider
|
|
26
31
|
@model = model
|
|
27
32
|
@batch_size = batch_size
|
|
28
33
|
@max_batch_chars = max_batch_chars
|
|
29
34
|
@pause = pause
|
|
30
35
|
@log = log
|
|
36
|
+
@accept = accept
|
|
31
37
|
end
|
|
32
38
|
|
|
33
39
|
# Translates `keys` (looked up in `source`) and returns
|
|
34
|
-
# `[translations, failed_keys]
|
|
40
|
+
# `[translations, failed_keys, rejections]`; `rejections` maps each failed
|
|
41
|
+
# key whose reply in its last round `accept` rejected to the reason.
|
|
35
42
|
def call(source, keys, instructions:)
|
|
36
43
|
@source = source
|
|
37
44
|
@instructions = instructions
|
|
38
45
|
@unparseable = []
|
|
46
|
+
@rejections = {}
|
|
39
47
|
|
|
40
48
|
translated, missing = translate_keys(keys)
|
|
41
49
|
missing = retry_missing(translated, missing)
|
|
42
50
|
|
|
43
51
|
failed = missing + @unparseable
|
|
44
52
|
report_failed(failed)
|
|
45
|
-
[translated, failed]
|
|
53
|
+
[translated, failed, @rejections]
|
|
46
54
|
end
|
|
47
55
|
|
|
48
56
|
private
|
|
@@ -75,7 +83,8 @@ module Locallingo
|
|
|
75
83
|
batches(keys).each do |batch|
|
|
76
84
|
result = translate_batch(batch.to_h { |key| [key, @source[key]] })
|
|
77
85
|
batch.each do |key|
|
|
78
|
-
|
|
86
|
+
@rejections.delete(key)
|
|
87
|
+
if result.key?(key) && !result[key].to_s.empty? && accepted?(key, result[key])
|
|
79
88
|
translations[key] = result[key]
|
|
80
89
|
else
|
|
81
90
|
failed << key
|
|
@@ -87,6 +96,17 @@ module Locallingo
|
|
|
87
96
|
[translations, failed]
|
|
88
97
|
end
|
|
89
98
|
|
|
99
|
+
def accepted?(key, value)
|
|
100
|
+
return true unless @accept
|
|
101
|
+
|
|
102
|
+
verdict = @accept.call(key, @source[key], value)
|
|
103
|
+
return true if verdict == true
|
|
104
|
+
|
|
105
|
+
@rejections[key] = verdict.is_a?(Hash) ? verdict : {}
|
|
106
|
+
log(" Rejected #{key}: #{@rejections[key]}", level: :warn)
|
|
107
|
+
false
|
|
108
|
+
end
|
|
109
|
+
|
|
90
110
|
def batches(keys)
|
|
91
111
|
return keys.each_slice(@batch_size).to_a unless @max_batch_chars
|
|
92
112
|
|
|
@@ -72,6 +72,13 @@ module Locallingo
|
|
|
72
72
|
def british_spellings? = !!dig("quality", "british_spellings")
|
|
73
73
|
def terminology_setting = dig("quality", "terminology")
|
|
74
74
|
|
|
75
|
+
# Token patterns (`translate.placeholders`, regex strings compiled with
|
|
76
|
+
# Regexp::MULTILINE, so `.` spans lines) whose matches a Translator must
|
|
77
|
+
# keep unchanged; [] by default.
|
|
78
|
+
def placeholders
|
|
79
|
+
Array(dig("translate", "placeholders")).map { |pattern| compile_placeholder(pattern) }.freeze
|
|
80
|
+
end
|
|
81
|
+
|
|
75
82
|
# --- prompt scaffolding --------------------------------------------------
|
|
76
83
|
|
|
77
84
|
def context = data.fetch("context", "a business application")
|
|
@@ -123,6 +130,17 @@ module Locallingo
|
|
|
123
130
|
|
|
124
131
|
def dig(*keys) = data.dig(*keys)
|
|
125
132
|
|
|
133
|
+
def compile_placeholder(pattern)
|
|
134
|
+
return pattern.dup.freeze if pattern.is_a?(Regexp)
|
|
135
|
+
unless pattern.is_a?(String)
|
|
136
|
+
raise Error, "translate.placeholders: expected a regex string, got #{pattern.inspect}"
|
|
137
|
+
end
|
|
138
|
+
|
|
139
|
+
Regexp.new(pattern, Regexp::MULTILINE).freeze
|
|
140
|
+
rescue RegexpError => e
|
|
141
|
+
raise Error, "translate.placeholders: #{pattern.to_s.inspect} is not a valid regular expression (#{e.message})"
|
|
142
|
+
end
|
|
143
|
+
|
|
126
144
|
# Merge: shipped defaults <- user `defaults:` <- selected `packages:` entry.
|
|
127
145
|
def resolve(source)
|
|
128
146
|
document, origin = user_config(source)
|
|
@@ -0,0 +1,35 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Locallingo
|
|
4
|
+
# Token checks for translated text: the tokens a list of patterns matches
|
|
5
|
+
# (`{{ item.name }}`, `%{count}`, …) must survive translation unchanged, in
|
|
6
|
+
# the same count. Order may change, since word order does. Pure functions
|
|
7
|
+
# over their arguments.
|
|
8
|
+
module Placeholders
|
|
9
|
+
module_function
|
|
10
|
+
|
|
11
|
+
# Every whole match of every pattern in +text+, pattern by pattern.
|
|
12
|
+
def tokens(text, patterns)
|
|
13
|
+
patterns.flat_map { |pattern| text.to_s.gsub(pattern).to_a }
|
|
14
|
+
end
|
|
15
|
+
|
|
16
|
+
# nil when +translated+ holds exactly the tokens of +source+, else
|
|
17
|
+
# `{ "missing" => [...], "extra" => [...] }` (multiset difference on the
|
|
18
|
+
# exact matched text).
|
|
19
|
+
def compare(source, translated, patterns)
|
|
20
|
+
before = tokens(source, patterns).tally
|
|
21
|
+
after = tokens(translated, patterns).tally
|
|
22
|
+
missing = difference(before, after)
|
|
23
|
+
extra = difference(after, before)
|
|
24
|
+
return nil if missing.empty? && extra.empty?
|
|
25
|
+
|
|
26
|
+
{ "missing" => missing, "extra" => extra }
|
|
27
|
+
end
|
|
28
|
+
|
|
29
|
+
def difference(counts, other)
|
|
30
|
+
counts.flat_map { |token, count| [token] * [count - other.fetch(token, 0), 0].max }
|
|
31
|
+
end
|
|
32
|
+
|
|
33
|
+
private_class_method :difference
|
|
34
|
+
end
|
|
35
|
+
end
|
data/lib/locallingo/prompt.rb
CHANGED
|
@@ -22,7 +22,7 @@ module Locallingo
|
|
|
22
22
|
|
|
23
23
|
module_function
|
|
24
24
|
|
|
25
|
-
def translation(config:, from:, to:, style: :ui, instructions: nil)
|
|
25
|
+
def translation(config:, from:, to:, style: :ui, instructions: nil, placeholders: [])
|
|
26
26
|
raise Error, "Unknown prompt style #{style.inspect} (expected one of #{STYLES.join(", ")})" unless
|
|
27
27
|
STYLES.include?(style.to_s.to_sym)
|
|
28
28
|
|
|
@@ -41,7 +41,7 @@ module Locallingo
|
|
|
41
41
|
|
|
42
42
|
## General Rules
|
|
43
43
|
- Preserve placeholders like #{config.placeholder_style} exactly as they appear
|
|
44
|
-
- Preserve HTML tags if present
|
|
44
|
+
#{placeholders_rule(placeholders)}- Preserve HTML tags if present
|
|
45
45
|
#{style.to_sym == :prose ? MARKDOWN_RULES : BREVITY_RULE}- Escape every double quote (") inside a translated value as \\" so the reply stays valid JSON
|
|
46
46
|
- Keep typographic quotes (“ ” „ ‚ « ») exactly as they appear in the source
|
|
47
47
|
#{glossary_section(config)}#{language_guide_section(config, to)}#{instructions_section(instructions)}
|
|
@@ -51,6 +51,14 @@ module Locallingo
|
|
|
51
51
|
PROMPT
|
|
52
52
|
end
|
|
53
53
|
|
|
54
|
+
def placeholders_rule(patterns)
|
|
55
|
+
patterns = Array(patterns)
|
|
56
|
+
return "" if patterns.empty?
|
|
57
|
+
|
|
58
|
+
sources = patterns.map { |pattern| "`#{pattern.source}`" }.join(", ")
|
|
59
|
+
"- Preserve every token matching these patterns exactly as it appears, with the same count: #{sources}\n"
|
|
60
|
+
end
|
|
61
|
+
|
|
54
62
|
def glossary_section(config)
|
|
55
63
|
return "" if config.glossary.empty?
|
|
56
64
|
|
|
@@ -68,6 +76,6 @@ module Locallingo
|
|
|
68
76
|
text.empty? ? "" : "\n## Additional Instructions\n#{text}\n"
|
|
69
77
|
end
|
|
70
78
|
|
|
71
|
-
private_class_method :glossary_section, :language_guide_section, :instructions_section
|
|
79
|
+
private_class_method :placeholders_rule, :glossary_section, :language_guide_section, :instructions_section
|
|
72
80
|
end
|
|
73
81
|
end
|
|
@@ -16,6 +16,9 @@ module Locallingo
|
|
|
16
16
|
# optional British-spelling drift, and a very-long-text heuristic. An optional
|
|
17
17
|
# AI pass (RubyLLM) reviews a sample for clarity/professionalism.
|
|
18
18
|
#
|
|
19
|
+
# `check` and `check_key` read the locale files; `check_texts` runs the same
|
|
20
|
+
# checks over a `{ key => text }` hash held anywhere else.
|
|
21
|
+
#
|
|
19
22
|
# `fix!` rewrites the auto-fixable suggestions (universal fixes + British
|
|
20
23
|
# spellings) back into the locale files, preserving case.
|
|
21
24
|
class QualityChecker
|
|
@@ -23,11 +26,13 @@ module Locallingo
|
|
|
23
26
|
|
|
24
27
|
attr_reader :config, :verbose, :logger
|
|
25
28
|
|
|
26
|
-
|
|
29
|
+
# +context+ is an optional RubyLLM::Context the AI pass runs on (e.g. a
|
|
30
|
+
# per-call key), as for Translator.
|
|
31
|
+
def initialize(config: nil, root_path: nil, package: nil, verbose: false, logger: nil, context: nil)
|
|
27
32
|
@config = config || Configuration.load(root_path: root_path || Dir.pwd, package:)
|
|
28
33
|
@verbose = verbose
|
|
29
34
|
@logger = logger
|
|
30
|
-
@provider = Providers::RubyLLM.new(provider: @config.provider)
|
|
35
|
+
@provider = Providers::RubyLLM.new(provider: @config.provider, context:)
|
|
31
36
|
@terminology = Quality::Terminology.new(@config.terminology_setting, base_path: @config.base_path)
|
|
32
37
|
end
|
|
33
38
|
|
|
@@ -37,20 +42,26 @@ module Locallingo
|
|
|
37
42
|
def check(locale: nil, use_ai: false, prefix: nil, keys: nil)
|
|
38
43
|
locale ||= config.source_locale
|
|
39
44
|
translations = scoped_translations(locale, prefix:, keys:)
|
|
45
|
+
return check_texts(translations, locale:, use_ai:) if keys || !use_ai
|
|
40
46
|
|
|
41
|
-
|
|
42
|
-
suggestions.concat(keys ? suggest_improvements(translations, locale) : ai_sample(translations, locale)) if use_ai
|
|
43
|
-
suggestions
|
|
47
|
+
check_texts(translations, locale:).concat(ai_sample(translations, locale))
|
|
44
48
|
end
|
|
45
49
|
|
|
46
50
|
# Check a single key.
|
|
47
51
|
def check_key(key, locale, use_ai: false)
|
|
48
|
-
|
|
49
|
-
text = translations[key]
|
|
52
|
+
text = load_locale_translations(locale)[key]
|
|
50
53
|
return [{ key:, error: "Key not found" }] unless text
|
|
51
54
|
|
|
52
|
-
|
|
53
|
-
|
|
55
|
+
check_texts({ key => text }, locale:, use_ai:)
|
|
56
|
+
end
|
|
57
|
+
|
|
58
|
+
# Check a `{ key => text }` hash without reading any file: the static
|
|
59
|
+
# checks on every pair, then, with +use_ai+, the AI pass over every given
|
|
60
|
+
# key (unsampled). Returns the suggestion hashes `check` returns.
|
|
61
|
+
def check_texts(texts, locale:, use_ai: false)
|
|
62
|
+
texts = stringify_keys(texts).select { |_key, text| text.is_a?(String) }
|
|
63
|
+
suggestions = texts.flat_map { |key, text| check_text(key, text, locale) }
|
|
64
|
+
suggestions.concat(suggest_improvements(texts, locale)) if use_ai && texts.any?
|
|
54
65
|
suggestions
|
|
55
66
|
end
|
|
56
67
|
|
|
@@ -105,6 +116,14 @@ module Locallingo
|
|
|
105
116
|
suggestions.compact
|
|
106
117
|
end
|
|
107
118
|
|
|
119
|
+
def stringify_keys(texts)
|
|
120
|
+
texts.each_with_object({}) do |(key, text), all|
|
|
121
|
+
raise Error, "Duplicate key #{key.to_s.inspect} (e.g. :#{key} and \"#{key}\")" if all.key?(key.to_s)
|
|
122
|
+
|
|
123
|
+
all[key.to_s] = text
|
|
124
|
+
end
|
|
125
|
+
end
|
|
126
|
+
|
|
108
127
|
def british_spellings_for?(locale)
|
|
109
128
|
config.british_spellings? && locale.to_s == config.source_locale.to_s
|
|
110
129
|
end
|
|
@@ -13,7 +13,8 @@ module Locallingo
|
|
|
13
13
|
# its row lock, after re-checking that its source is unchanged and the
|
|
14
14
|
# target is still due: a source edited or a target hand-written while the
|
|
15
15
|
# model answered is skipped and left for the next run. A record with any
|
|
16
|
-
# failed segment
|
|
16
|
+
# failed segment (including one whose tokens did not survive, with
|
|
17
|
+
# `placeholders:`) is not written; it stays due and its paths are reported.
|
|
17
18
|
#
|
|
18
19
|
# Records must be persisted and of one model (ids are the key prefix). This
|
|
19
20
|
# class never loads ActiveRecord itself; records only need the concern.
|
|
@@ -30,7 +31,8 @@ module Locallingo
|
|
|
30
31
|
@write = write
|
|
31
32
|
end
|
|
32
33
|
|
|
33
|
-
|
|
34
|
+
# +placeholders+ is passed to Translator#translate (nil: its default).
|
|
35
|
+
def call(records, to:, style: :ui, instructions: nil, force: false, include_manual: false, placeholders: nil)
|
|
34
36
|
@locale = to.to_s
|
|
35
37
|
@force = force
|
|
36
38
|
@include_manual = include_manual
|
|
@@ -39,7 +41,7 @@ module Locallingo
|
|
|
39
41
|
due, empty = records.select { |record| due?(record) }.partition { |record| record.translation_segments.any? }
|
|
40
42
|
@result[:skipped].concat(empty.map(&:id))
|
|
41
43
|
due.group_by(&:translation_source_locale).each do |from, group|
|
|
42
|
-
translate_group(group, from:, style:, instructions:)
|
|
44
|
+
translate_group(group, from:, style:, instructions:, placeholders:)
|
|
43
45
|
end
|
|
44
46
|
@result
|
|
45
47
|
end
|
|
@@ -48,7 +50,7 @@ module Locallingo
|
|
|
48
50
|
|
|
49
51
|
def due?(record) = record.translation_due?(@locale, force: @force, include_manual: @include_manual)
|
|
50
52
|
|
|
51
|
-
def translate_group(records, from:, style:, instructions:)
|
|
53
|
+
def translate_group(records, from:, style:, instructions:, placeholders:)
|
|
52
54
|
digests = records.to_h { |record| [record, record.translation_source_digest] }
|
|
53
55
|
owners = {}
|
|
54
56
|
segments = records.each_with_object({}) do |record, all|
|
|
@@ -61,7 +63,7 @@ module Locallingo
|
|
|
61
63
|
end
|
|
62
64
|
end
|
|
63
65
|
|
|
64
|
-
result = @translator.translate(segments, from:, to: @locale, style:, instructions:)
|
|
66
|
+
result = @translator.translate(segments, from:, to: @locale, style:, instructions:, placeholders:)
|
|
65
67
|
records.each do |record|
|
|
66
68
|
translations = own(result.translations, owners, record).to_h
|
|
67
69
|
failed = own(result.failed.to_h { |key| [key, nil] }, owners, record).map(&:first)
|
|
@@ -6,6 +6,7 @@ require_relative "configuration"
|
|
|
6
6
|
require_relative "prompt"
|
|
7
7
|
require_relative "providers/ruby_llm"
|
|
8
8
|
require_relative "batch_translator"
|
|
9
|
+
require_relative "placeholders"
|
|
9
10
|
|
|
10
11
|
module Locallingo
|
|
11
12
|
# The public, file-agnostic entry point: translates a plain `{ key => text }`
|
|
@@ -19,11 +20,19 @@ module Locallingo
|
|
|
19
20
|
# result = translator.translate({ "title" => "Hello" }, from: "en", to: "de", style: :prose)
|
|
20
21
|
# result.translations # => { "title" => "Hallo" }
|
|
21
22
|
# result.failed # => []
|
|
23
|
+
# result.rejections # => {}
|
|
22
24
|
#
|
|
23
25
|
# Batching, retries, batch halving and omitted-key rounds come from
|
|
24
26
|
# BatchTranslator. Blank texts are not sent.
|
|
27
|
+
#
|
|
28
|
+
# With `placeholders:` (Regexps; default `translate.placeholders`), a value
|
|
29
|
+
# whose tokens differ from its source's is re-requested in those rounds and,
|
|
30
|
+
# if it still differs, reported in `failed` and `rejections` instead of
|
|
31
|
+
# returned.
|
|
25
32
|
class Translator
|
|
26
|
-
Result = Data.define(:translations, :failed)
|
|
33
|
+
Result = Data.define(:translations, :failed, :rejections) do
|
|
34
|
+
def initialize(translations:, failed:, rejections: {}) = super
|
|
35
|
+
end
|
|
27
36
|
|
|
28
37
|
attr_reader :config
|
|
29
38
|
|
|
@@ -38,15 +47,16 @@ module Locallingo
|
|
|
38
47
|
end
|
|
39
48
|
|
|
40
49
|
# Raises MissingCredentialsError before any call when no key is available.
|
|
41
|
-
def translate(texts, from:, to:, style: :ui, instructions: nil)
|
|
50
|
+
def translate(texts, from:, to:, style: :ui, instructions: nil, placeholders: nil)
|
|
51
|
+
patterns = placeholder_patterns(placeholders)
|
|
42
52
|
@provider.ensure_credentials!
|
|
43
53
|
|
|
44
54
|
source = stringify_keys(texts).reject { |_key, text| text.to_s.strip.empty? }
|
|
45
|
-
return Result.new(translations: {}, failed: []) if source.empty?
|
|
55
|
+
return Result.new(translations: {}, failed: [], rejections: {}) if source.empty?
|
|
46
56
|
|
|
47
|
-
prompt = Prompt.translation(config:, from:, to:, style:, instructions:)
|
|
48
|
-
translated, failed = batch_translator.call(source, source.keys, instructions: prompt)
|
|
49
|
-
Result.new(translations: translated.slice(*source.keys).except(*failed), failed:)
|
|
57
|
+
prompt = Prompt.translation(config:, from:, to:, style:, instructions:, placeholders: patterns)
|
|
58
|
+
translated, failed, rejections = batch_translator(patterns).call(source, source.keys, instructions: prompt)
|
|
59
|
+
Result.new(translations: translated.slice(*source.keys).except(*failed), failed:, rejections:)
|
|
50
60
|
end
|
|
51
61
|
|
|
52
62
|
private
|
|
@@ -59,12 +69,28 @@ module Locallingo
|
|
|
59
69
|
end
|
|
60
70
|
end
|
|
61
71
|
|
|
62
|
-
def
|
|
63
|
-
|
|
72
|
+
def placeholder_patterns(placeholders)
|
|
73
|
+
patterns = placeholders.nil? ? config.placeholders : Array(placeholders)
|
|
74
|
+
invalid = patterns.grep_v(Regexp)
|
|
75
|
+
raise Error, "placeholders: expected Regexps, got #{invalid.map(&:inspect).join(", ")}" if invalid.any?
|
|
76
|
+
|
|
77
|
+
patterns
|
|
78
|
+
end
|
|
79
|
+
|
|
80
|
+
# One per call: `accept` closes over that call's patterns.
|
|
81
|
+
def batch_translator(patterns)
|
|
82
|
+
BatchTranslator.new(
|
|
64
83
|
provider: @provider, model: @model, batch_size: config.batch_size,
|
|
65
84
|
max_batch_chars: @max_batch_chars, pause: @pause,
|
|
66
|
-
log: ->(message, level: :info) { @logger.public_send(level, message) }
|
|
85
|
+
log: ->(message, level: :info) { @logger.public_send(level, message) },
|
|
86
|
+
accept: accept_for(patterns)
|
|
67
87
|
)
|
|
68
88
|
end
|
|
89
|
+
|
|
90
|
+
def accept_for(patterns)
|
|
91
|
+
return nil if patterns.empty?
|
|
92
|
+
|
|
93
|
+
->(_key, source_text, translated) { Placeholders.compare(source_text, translated, patterns) || true }
|
|
94
|
+
end
|
|
69
95
|
end
|
|
70
96
|
end
|
data/lib/locallingo/version.rb
CHANGED
data/lib/locallingo.rb
CHANGED
|
@@ -9,6 +9,7 @@ require_relative "locallingo/key_flattener"
|
|
|
9
9
|
require_relative "locallingo/state_store"
|
|
10
10
|
require_relative "locallingo/drift"
|
|
11
11
|
require_relative "locallingo/providers/ruby_llm"
|
|
12
|
+
require_relative "locallingo/placeholders"
|
|
12
13
|
require_relative "locallingo/batch_translator"
|
|
13
14
|
require_relative "locallingo/prompt"
|
|
14
15
|
require_relative "locallingo/translator"
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: locallingo
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.6.
|
|
4
|
+
version: 0.6.1
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Mikael Henriksson
|
|
@@ -78,6 +78,7 @@ files:
|
|
|
78
78
|
- lib/locallingo/json_extraction.rb
|
|
79
79
|
- lib/locallingo/key_flattener.rb
|
|
80
80
|
- lib/locallingo/manager.rb
|
|
81
|
+
- lib/locallingo/placeholders.rb
|
|
81
82
|
- lib/locallingo/prompt.rb
|
|
82
83
|
- lib/locallingo/providers/ruby_llm.rb
|
|
83
84
|
- lib/locallingo/quality/british_spellings.rb
|