i18n-tasks 1.0.13 → 1.0.14

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 12e19e4d7fe61bd20e000b0b6e4db5bdf233443e8e8cf965b62e634bdfa3e5a2
4
- data.tar.gz: 3dab05338d079c4defad590466ab2985faf43245d616f86af4c212003831a6e4
3
+ metadata.gz: bd61ceb254dc1c44768bd40672b3f31635c9d42fb237950cbdfe64fef642a850
4
+ data.tar.gz: ae6981dd739636ddb5f1bae1ca9e5cf05ba991fd1345a5488e1cfa8561833877
5
5
  SHA512:
6
- metadata.gz: 2cf99aec1c6d5738c4a71964a7ce67e70c8ade3150042c4e381fef19b9eaa82cf6638979bf97c6e9941f119304973f9e6f3fd31b3bab0fd635520344448b5587
7
- data.tar.gz: 2de5ab47951585b298e43bece202f3ad885ae8eb03edb2b6e860b30e3172ea213947509b9543bbf31df06ef03079ff1e7cfd1b7a32cd964428d9f9612fd7e171
6
+ metadata.gz: 1312910139411cc4695bea0f0de8624d49958efac33c8c7b7bd99fb3f351d408d0a6980789ec81a43452bc29a15292f6de9719ace0e3b902866d63c02f5cab79
7
+ data.tar.gz: e782f1bc1f1ad68cc33583d014c879b3d121ba2e0990db8cc109f864aeb0e98f0a97030300294e31e9041899b131b2b677fe6917f705d09bda6bf07aed588b42
data/README.md CHANGED
@@ -24,7 +24,7 @@ i18n-tasks can be used with any project using the ruby [i18n gem][i18n-gem] (def
24
24
  Add i18n-tasks to the Gemfile:
25
25
 
26
26
  ```ruby
27
- gem 'i18n-tasks', '~> 1.0.13'
27
+ gem 'i18n-tasks', '~> 1.0.14', group: :development
28
28
  ```
29
29
 
30
30
  Copy the default [configuration file](#configuration):
@@ -338,6 +338,33 @@ data:
338
338
  If you want to have i18n-tasks reorganize your existing keys using `data.write`, either set the router to
339
339
  `pattern_router` as above, or run `i18n-tasks normalize -p` (forcing the use of the pattern router for that run).
340
340
 
341
+ ##### Isolating router
342
+
343
+ Isolating router assumes each YAML file is independent and can contain similar keys.
344
+
345
+ As a result, the translations are written to an alternate target file for each source file
346
+ (only the `%{locale}` part is changed to match target locale). Thus, it is not necessary to
347
+ specify any `write` configuration (in fact, it would be completely ignored).
348
+
349
+ This can be useful for example when using [ViewComponent sidecars](https://viewcomponent.org/guide/translations.html)
350
+ (ViewComponent assigns an implicit scope to each sidecar YAML file but `i18n-tasks` is not aware of
351
+ that logic, resulting in collisions):
352
+
353
+ * `app/components/movies_component.en.yml`:
354
+ ```yaml
355
+ en:
356
+ title: Movies
357
+ ```
358
+
359
+ * `app/components/games_component.en.yml`
360
+ ```yaml
361
+ en:
362
+ title: Games
363
+ ```
364
+
365
+ This router has a limitation, though: it does not support detecting missing keys from code usage
366
+ (since it is not aware of the implicit scope logic).
367
+
341
368
  ##### Key pattern syntax
342
369
 
343
370
  A special syntax similar to file glob patterns is used throughout i18n-tasks to match translation keys:
@@ -411,6 +438,12 @@ translation:
411
438
  google_translate_api_key: <Google Translate API key>
412
439
  ```
413
440
 
441
+ or via environment variable:
442
+
443
+ ```bash
444
+ GOOGLE_TRANSLATE_API_KEY=<Google Translate API key>
445
+ ```
446
+
414
447
  <a name="deepl-translation-config"></a>
415
448
  ### DeepL Pro Translate
416
449
 
@@ -422,6 +455,19 @@ translation:
422
455
  deepl_api_key: <DeepL Pro API key>
423
456
  deepl_host: <optional>
424
457
  deepl_version: <optional>
458
+ deepl_glossary_ids:
459
+ - f28106eb-0e06-489e-82c6-8215d6f95089
460
+ - 2c6415be-1852-4f54-9e1b-d800463496b4
461
+ deepl_options:
462
+ formality: prefer_less
463
+ ```
464
+
465
+ or via environment variables:
466
+
467
+ ```bash
468
+ DEEPL_API_KEY=<DeepL Pro API key>
469
+ DEEPL_HOST=<optional>
470
+ DEEPL_VERSION=<optional>
425
471
  ```
426
472
 
427
473
  <a name="yandex-translation-config"></a>
@@ -435,6 +481,12 @@ translation:
435
481
  yandex_api_key: <Yandex API key>
436
482
  ```
437
483
 
484
+ or via environment variable:
485
+
486
+ ```bash
487
+ YANDEX_API_KEY=<Yandex API key>
488
+ ```
489
+
438
490
  <a name="openai-translation-config"></a>
439
491
  ### OpenAI Translate
440
492
 
@@ -444,6 +496,14 @@ translation:
444
496
  # config/i18n-tasks.yml
445
497
  translation:
446
498
  openai_api_key: <OpenAI API key>
499
+ openai_model: <optional>
500
+ ```
501
+
502
+ or via environment variable:
503
+
504
+ ```bash
505
+ OPENAI_API_KEY=<OpenAI API key>
506
+ OPENAI_MODEL=<optional>
447
507
  ```
448
508
 
449
509
  ## Interactive console
@@ -16,12 +16,10 @@ en:
16
16
  data_format: 'Data format: %{valid_text}.'
17
17
  keep_order: Keep the order of the keys
18
18
  key_pattern: Filter by key pattern (e.g. 'common.*')
19
- key_pattern_to_rename: Full key (pattern) to rename. Required
20
19
  locale: :i18n_tasks.common.locale
21
20
  locale_to_translate_from: Locale to translate from
22
21
  locales_filter: 'Locale(s) to process. Special: base'
23
22
  missing_types: 'Filter by types: %{valid}'
24
- new_key_name: New name, interpolates original name as %{key}. Required
25
23
  nostdin: Do not read from stdin
26
24
  out_format: 'Output format: %{valid_text}'
27
25
  pattern_router: 'Use pattern router: keys moved per config data.write'
@@ -30,13 +28,14 @@ en:
30
28
  the config setting if set.
31
29
  translation_backend: Translation backend (google or deepl)
32
30
  value: >-
33
- Value. Interpolates: %{value}, %{human_key}, %{key}, %{default}, %{value_or_human_key},
34
- %{value_or_default_or_human_key}
31
+ Value. Interpolates: %%{value}, %%{human_key}, %%{key}, %%{default}, %%{value_or_human_key},
32
+ %%{value_or_default_or_human_key}
35
33
  desc:
36
34
  add_missing: add missing keys to locale data, optionally match a pattern
37
35
  check_consistent_interpolations: verify that all translations use correct interpolation variables
38
36
  check_normalized: verify that all translation data is normalized
39
37
  config: display i18n-tasks configuration
38
+ cp: copy the keys in locale data that match the given pattern
40
39
  data: show locale data
41
40
  data_merge: merge locale data with trees
42
41
  data_remove: remove keys present in tree from data
@@ -13,28 +13,27 @@ ru:
13
13
  data_format: 'Формат данных: %{valid_text}.'
14
14
  keep_order: Keep the order of the keys
15
15
  key_pattern: Маска ключа (например, common.*)
16
- key_pattern_to_rename: Полный ключ (шаблон) для переименования. Необходимый параметр.
17
16
  locale: 'Язык. По умолчанию: base'
18
17
  locale_to_translate_from: 'Язык, с которого переводить (по умолчанию: base)'
19
18
  locales_filter: >-
20
19
  Список языков для обработки, разделенный запятыми (,). По умолчанию: все. Специальное
21
20
  значение: base.
22
21
  missing_types: 'Типы недостающих переводов: %{valid}. По умолчанию: все'
23
- new_key_name: Новое имя, интерполирует оригинальное название как %{key}. Необходимый параметр.
24
22
  nostdin: Не читать дерево из стандартного ввода
25
23
  out_format: 'Формат вывода: %{valid_text}.'
26
24
  pattern_router: 'Использовать pattern_router: ключи распределятся по файлам согласно data.write'
27
25
  strict: Не угадывать динамические использования ключей, например `t("category.#{category.key}")`
28
26
  translation_backend: Движок перевода (google или deepl)
29
27
  value: >-
30
- Значение, интерполируется с %{value}, %{human_key}, %{key}, %{default}, %{value_or_human_key},
31
- %{value_or_default_or_human_key}
28
+ Значение, интерполируется с %%{value}, %%{human_key}, %%{key}, %%{default}, %%{value_or_human_key},
29
+ %%{value_or_default_or_human_key}
32
30
  desc:
33
31
  add_missing: добавить недостающие ключи к переводам
34
32
  check_consistent_interpolations: убедитесь, что во всех переводах используются правильные
35
33
  интерполяционные переменные
36
34
  check_normalized: проверить, что все файлы переводов нормализованы
37
35
  config: показать конфигурацию
36
+ cp: скопируйте ключи в данных локали, соответствующие заданному шаблону
38
37
  data: показать данные переводов
39
38
  data_merge: добавить дерево к переводам
40
39
  data_remove: удалить ключи, которые есть в дереве, из данных
data/i18n-tasks.gemspec CHANGED
@@ -39,7 +39,6 @@ Gem::Specification.new do |s|
39
39
 
40
40
  s.add_dependency 'activesupport', '>= 4.0.2'
41
41
  s.add_dependency 'ast', '>= 2.1.0'
42
- s.add_dependency 'better_html', '>= 1.0', '< 3.0'
43
42
  s.add_dependency 'erubi'
44
43
  s.add_dependency 'highline', '>= 2.0.0'
45
44
  s.add_dependency 'i18n'
@@ -46,6 +46,20 @@ module I18n::Tasks
46
46
  terminal_report.mv_results results
47
47
  end
48
48
 
49
+ cmd :cp,
50
+ pos: 'FROM_KEY_PATTERN TO_KEY_PATTERN',
51
+ desc: t('i18n_tasks.cmd.desc.cp')
52
+ def cp(opt = {})
53
+ fail CommandError, 'requires FROM_KEY_PATTERN and TO_KEY_PATTERN' if opt[:arguments].size < 2
54
+
55
+ from_pattern = opt[:arguments].shift
56
+ to_pattern = opt[:arguments].shift
57
+ forest = i18n.data_forest
58
+ results = forest.mv_key!(compile_key_pattern(from_pattern), to_pattern, root: false, retain: true)
59
+ i18n.data.write forest
60
+ terminal_report.cp_results results
61
+ end
62
+
49
63
  cmd :rm,
50
64
  pos: 'KEY_PATTERN [KEY_PATTERN...]',
51
65
  desc: t('i18n_tasks.cmd.desc.rm')
@@ -26,7 +26,7 @@ module I18n::Tasks
26
26
  arg :value,
27
27
  '-v',
28
28
  '--value VALUE',
29
- t('i18n_tasks.cmd.args.desc.value')
29
+ t('i18n_tasks.cmd.args.desc.value', dummy: 'value') # Dummy value is workaround for https://github.com/ruby-i18n/i18n/issues/689
30
30
 
31
31
  arg :config,
32
32
  '-c',
@@ -67,6 +67,7 @@ module I18n::Tasks::Configuration # rubocop:disable Metrics/ModuleLength
67
67
  conf[:deepl_host] = ENV['DEEPL_HOST'] if ENV.key?('DEEPL_HOST')
68
68
  conf[:deepl_version] = ENV['DEEPL_VERSION'] if ENV.key?('DEEPL_VERSION')
69
69
  conf[:openai_api_key] = ENV['OPENAI_API_KEY'] if ENV.key?('OPENAI_API_KEY')
70
+ conf[:openai_model] = ENV['OPENAI_MODEL'] if ENV.key?('OPENAI_MODEL')
70
71
  conf[:yandex_api_key] = ENV['YANDEX_API_KEY'] if ENV.key?('YANDEX_API_KEY')
71
72
  conf
72
73
  end
@@ -3,6 +3,7 @@
3
3
  require 'i18n/tasks/data/tree/node'
4
4
  require 'i18n/tasks/data/router/pattern_router'
5
5
  require 'i18n/tasks/data/router/conservative_router'
6
+ require 'i18n/tasks/data/router/isolating_router'
6
7
  require 'i18n/tasks/data/file_formats'
7
8
  require 'i18n/tasks/key_pattern_matching'
8
9
 
@@ -151,6 +152,7 @@ module I18n::Tasks
151
152
 
152
153
  ROUTER_NAME_ALIASES = {
153
154
  'conservative_router' => 'I18n::Tasks::Data::Router::ConservativeRouter',
155
+ 'isolating_router' => 'I18n::Tasks::Data::Router::IsolatingRouter',
154
156
  'pattern_router' => 'I18n::Tasks::Data::Router::PatternRouter'
155
157
  }.freeze
156
158
  def router
@@ -170,6 +172,9 @@ module I18n::Tasks
170
172
  end.map do |path|
171
173
  [path.freeze, load_file(path) || {}]
172
174
  end.map do |path, data|
175
+ if router.is_a?(I18n::Tasks::Data::Router::IsolatingRouter)
176
+ data.transform_values! { |tree| { "<#{router.alternate_path_for(path, base_locale)}>" => tree } }
177
+ end
173
178
  filter_nil_keys! path, data
174
179
  Data::Tree::Siblings.from_nested_hash(data).tap do |s|
175
180
  s.leaves { |x| x.data.update(path: path, locale: locale) }
@@ -0,0 +1,146 @@
1
+ # frozen_string_literal: true
2
+
3
+ require 'i18n/tasks/key_pattern_matching'
4
+ require 'i18n/tasks/data/tree/node'
5
+
6
+ module I18n::Tasks
7
+ module Data::Router
8
+ # Route based on source file path
9
+ class IsolatingRouter
10
+ include ::I18n::Tasks::KeyPatternMatching
11
+
12
+ attr_reader :config_read_patterns, :base_locale
13
+
14
+ def initialize(_adapter, data_config)
15
+ @base_locale = data_config[:base_locale]
16
+ @config_read_patterns = Array.wrap(data_config[:read])
17
+ end
18
+
19
+ # Route keys to destinations
20
+ # @param forest [I18n::Tasks::Data::Tree::Siblings] forest roots are locales.
21
+ # @yieldparam [String] dest_path
22
+ # @yieldparam [I18n::Tasks::Data::Tree::Siblings] tree_slice
23
+ # @return [Hash] mapping of destination => [ [key, value], ... ]
24
+ def route(locale, forest, &block)
25
+ return to_enum(:route, locale, forest) unless block
26
+
27
+ locale = locale.to_s
28
+ out = {}
29
+
30
+ forest.keys do |key_namespaced_with_source_path, _node|
31
+ source_path, key = key_namespaced_with_source_path.match(/\A<([^>]*)>\.(.*)/).captures
32
+ target_path = alternate_path_for(source_path, locale)
33
+ next unless source_path && key && target_path
34
+
35
+ (out[target_path] ||= Set.new) << "#{locale}.#{key}"
36
+ end
37
+
38
+ out.each do |target_path, keys|
39
+ file_namespace_subtree = I18n::Tasks::Data::Tree::Siblings.new(
40
+ nodes: forest.get("#{locale}.<#{alternate_path_for(target_path, base_locale)}>")
41
+ )
42
+ file_namespace_subtree.set_root_key!(locale)
43
+
44
+ block.yield(
45
+ target_path,
46
+ file_namespace_subtree.select_keys { |key, _| keys.include?(key) }
47
+ )
48
+ end
49
+ end
50
+
51
+ def alternate_path_for(source_path, locale)
52
+ source_path = source_path.dup
53
+
54
+ config_read_patterns.each do |pattern|
55
+ regexp = Glob.new(format(pattern, locale: '(*)')).to_regexp
56
+ next unless source_path.match?(regexp)
57
+
58
+ source_path.match(regexp) do |match_data|
59
+ (1..match_data.size - 1).reverse_each do |capture_index|
60
+ capture_begin, capture_end = match_data.offset(capture_index)
61
+ source_path.slice!(Range.new(capture_begin, capture_end, true))
62
+ source_path.insert(capture_begin, locale.to_s)
63
+ end
64
+ end
65
+
66
+ return source_path
67
+ end
68
+
69
+ nil
70
+ end
71
+
72
+ # based on https://github.com/alexch/rerun/blob/36f2d237985b670752abbe4a7f6814893cdde96f/lib/rerun/glob.rb
73
+ class Glob
74
+ NO_LEADING_DOT = '(?=[^\.])'
75
+ START_OF_FILENAME = '(?:\A|\/)'
76
+ END_OF_STRING = '\z'
77
+
78
+ def initialize(pattern)
79
+ @pattern = pattern
80
+ end
81
+
82
+ def to_regexp_string # rubocop:disable Metrics/CyclomaticComplexity, Metrics/MethodLength
83
+ chars = smoosh(@pattern.chars)
84
+
85
+ curlies = 0
86
+ escaping = false
87
+
88
+ string = chars.map do |char|
89
+ if escaping
90
+ escaping = false
91
+ next char
92
+ end
93
+
94
+ case char
95
+ when '**' then '(?:[^/]+/)*'
96
+ when '*' then '.*'
97
+ when '?' then '.'
98
+ when '.' then '\.'
99
+ when '{'
100
+ curlies += 1
101
+ '('
102
+ when '}'
103
+ if curlies.positive?
104
+ curlies -= 1
105
+ ')'
106
+ else
107
+ char
108
+ end
109
+ when ','
110
+ if curlies.positive?
111
+ '|'
112
+ else
113
+ char
114
+ end
115
+ when '\\'
116
+ escaping = true
117
+ '\\'
118
+ else char
119
+ end
120
+ end.join
121
+
122
+ START_OF_FILENAME + string + END_OF_STRING
123
+ end
124
+
125
+ def to_regexp
126
+ Regexp.new(to_regexp_string)
127
+ end
128
+
129
+ def smoosh(chars)
130
+ out = []
131
+ until chars.empty?
132
+ char = chars.shift
133
+ if char == '*' && chars.first == '*'
134
+ chars.shift
135
+ chars.shift if chars.first == '/'
136
+ out.push('**')
137
+ else
138
+ out.push(char)
139
+ end
140
+ end
141
+ out
142
+ end
143
+ end
144
+ end
145
+ end
146
+ end
@@ -36,7 +36,7 @@ module I18n::Tasks::Data::Tree
36
36
  # @param to_pattern [Regexp]
37
37
  # @param root [Boolean]
38
38
  # @return {old key => new key}
39
- def mv_key!(from_pattern, to_pattern, root: false) # rubocop:disable Metrics/AbcSize,Metrics/MethodLength
39
+ def mv_key!(from_pattern, to_pattern, root: false, retain: false) # rubocop:disable Metrics/AbcSize,Metrics/MethodLength
40
40
  moved_forest = Siblings.new
41
41
  moved_nodes = []
42
42
  old_key_to_new_key = {}
@@ -69,7 +69,7 @@ module I18n::Tasks::Data::Tree
69
69
  node.value = new_target.to_sym
70
70
  end
71
71
  end
72
- remove_nodes_and_emptied_ancestors! moved_nodes
72
+ remove_nodes_and_emptied_ancestors!(moved_nodes) unless retain
73
73
  merge! moved_forest
74
74
  old_key_to_new_key
75
75
  end
@@ -5,7 +5,7 @@ module I18n::Tasks
5
5
  class << self
6
6
  attr_accessor :variable_regex
7
7
  end
8
- @variable_regex = /%{[^}]+}/.freeze
8
+ @variable_regex = /(?<!%)%{[^}]+}/.freeze
9
9
 
10
10
  def inconsistent_interpolations(locales: nil, base_locale: nil) # rubocop:disable Metrics/AbcSize
11
11
  locales ||= self.locales
@@ -31,10 +31,10 @@ module I18n::Tasks::KeyPatternMatching
31
31
 
32
32
  def key_pattern_re_body(key_pattern)
33
33
  key_pattern
34
- .gsub(/\./, '\.')
35
- .gsub(/\*:/, '[^.]+?')
36
- .gsub(/\*/, '.*')
37
- .gsub(/:/, '(?<=^|\.)[^.]+?(?=\.|$)')
34
+ .gsub('.', '\.')
35
+ .gsub('*:', '[^.]+?')
36
+ .gsub('*', '.*')
37
+ .gsub(':', '(?<=^|\.)[^.]+?(?=\.|$)')
38
38
  .gsub(/\{(.*?)}/) { "(#{Regexp.last_match(1).strip.gsub(/\s*,\s*/, '|')})" }
39
39
  end
40
40
  end
@@ -93,6 +93,12 @@ module I18n
93
93
  end
94
94
  end
95
95
 
96
+ def cp_results(results)
97
+ results.each do |(from, to)|
98
+ print_info "#{Rainbow(from).cyan} #{Rainbow('+').yellow.bright} #{Rainbow(to).green}"
99
+ end
100
+ end
101
+
96
102
  def check_normalized_results(non_normalized)
97
103
  if non_normalized.empty?
98
104
  print_success 'All data is normalized'
@@ -0,0 +1,26 @@
1
+ # frozen_string_literal: true
2
+
3
+ require 'i18n/tasks/scanners/ast_matchers/base_matcher'
4
+ require 'i18n/tasks/scanners/results/occurrence'
5
+
6
+ module I18n::Tasks::Scanners::AstMatchers
7
+ class DefaultI18nSubjectMatcher < BaseMatcher
8
+ def convert_to_key_occurrences(send_node, method_name, location: send_node.loc)
9
+ children = Array(send_node&.children)
10
+ return unless children[1] == :default_i18n_subject
11
+
12
+ key = @scanner.absolute_key(
13
+ '.subject',
14
+ location.expression.source_buffer.name,
15
+ calling_method: method_name
16
+ )
17
+ [
18
+ key,
19
+ I18n::Tasks::Scanners::Results::Occurrence.from_range(
20
+ raw_key: key,
21
+ range: location.expression
22
+ )
23
+ ]
24
+ end
25
+ end
26
+ end
@@ -1,16 +1,16 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  require 'i18n/tasks/scanners/ruby_ast_scanner'
4
- require 'i18n/tasks/scanners/erb_ast_processor'
5
- require 'better_html/errors'
6
- require 'better_html/parser'
4
+ require 'i18n/tasks/scanners/local_ruby_parser'
7
5
 
8
6
  module I18n::Tasks::Scanners
9
7
  # Scan for I18n.translate calls in ERB-file better-html and ASTs
10
8
  class ErbAstScanner < RubyAstScanner
9
+ DEFAULT_REGEXP = /<%(={1,2}|-|\#|%)?(.*?)([-=])?%>/m.freeze
10
+
11
11
  def initialize(**args)
12
12
  super(**args)
13
- @erb_ast_processor = ErbAstProcessor.new
13
+ @ruby_parser = LocalRubyParser.new(ignore_blocks: true)
14
14
  end
15
15
 
16
16
  private
@@ -20,29 +20,59 @@ module I18n::Tasks::Scanners
20
20
  # @param path Path to file to parse
21
21
  # @return [{Parser::AST::Node}, [Parser::Source::Comment]]
22
22
  def path_to_ast_and_comments(path)
23
- parser = BetterHtml::Parser.new(make_buffer(path))
24
- ast = convert_better_html(parser.ast)
25
- @erb_ast_processor.process_and_extract_comments(ast)
23
+ comments = []
24
+ buffer = make_buffer(path)
25
+
26
+ children = []
27
+ buffer
28
+ .source
29
+ .scan(DEFAULT_REGEXP) do |indicator, code, tailch, _rspace|
30
+ match = Regexp.last_match
31
+ character = indicator ? indicator[0] : nil
32
+
33
+ start = match.begin(0) + 2 + (character&.size || 0)
34
+ stop = match.end(0) - 2 - (tailch&.size || 0)
35
+
36
+ case character
37
+ when '=', nil, '-'
38
+ parsed, parsed_comments = handle_code(buffer, code, start, stop)
39
+ comments.concat(parsed_comments)
40
+ children << parsed unless parsed.nil?
41
+ when '#', '#-'
42
+ comments << handle_comment(buffer, start, stop)
43
+ end
44
+ end
45
+
46
+ [root_node(children, buffer), comments]
26
47
  end
27
48
 
28
- # Convert BetterHtml nodes to Parser::AST::Node
29
- #
30
- # @param node BetterHtml::Parser::AST::Node
31
- # @return Parser::AST::Node
32
- def convert_better_html(node)
33
- definition = Parser::Source::Map::Definition.new(
34
- node.location.begin,
35
- node.location.begin,
36
- node.location.begin,
37
- node.location.end
38
- )
39
- Parser::AST::Node.new(
40
- node.type,
41
- node.children.map { |child| child.is_a?(BetterHtml::AST::Node) ? convert_better_html(child) : child },
42
- {
43
- location: definition
44
- }
45
- )
49
+ def handle_code(buffer, code, start, stop)
50
+ range = ::Parser::Source::Range.new(buffer, start, stop)
51
+ location =
52
+ Parser::Source::Map::Definition.new(
53
+ range.begin,
54
+ range.begin,
55
+ range.begin,
56
+ range.end
57
+ )
58
+ @ruby_parser.parse(code, location: location)
59
+ end
60
+
61
+ def handle_comment(buffer, start, stop)
62
+ range = ::Parser::Source::Range.new(buffer, start, stop)
63
+ ::Parser::Source::Comment.new(range)
64
+ end
65
+
66
+ def root_node(children, buffer)
67
+ range = ::Parser::Source::Range.new(buffer, 0, buffer.source.size)
68
+ location =
69
+ Parser::Source::Map::Definition.new(
70
+ range.begin,
71
+ range.begin,
72
+ range.begin,
73
+ range.end
74
+ )
75
+ ::Parser::AST::Node.new(:erb, children, location: location)
46
76
  end
47
77
  end
48
78
  end
@@ -12,7 +12,7 @@ module I18n::Tasks::Scanners
12
12
  include OccurrenceFromPosition
13
13
  include RubyKeyLiterals
14
14
 
15
- TRANSLATE_CALL_RE = /(?<=^|[^\w'\-.]|[^\w'-]I18n\.|I18n\.)t(?:!|ranslate!?)?/.freeze
15
+ TRANSLATE_CALL_RE = /(?<=^|[^\p{L}'\-.]|[^\p{L}'-]I18n\.|I18n\.)t(?:!|ranslate!?)?/.freeze
16
16
  IGNORE_LINES = {
17
17
  'coffee' => /^\s*#(?!\si18n-tasks-use)/,
18
18
  'erb' => /^\s*<%\s*#(?!\si18n-tasks-use)/,
@@ -61,7 +61,7 @@ module I18n
61
61
  "#{file_key.sub(/_controller$/, '')}.#{calling_method}"
62
62
  else
63
63
  # Remove _ prefix from partials
64
- file_key.gsub(/\._/, DOT)
64
+ file_key.gsub('._', DOT)
65
65
  end
66
66
  end
67
67
  end
@@ -3,6 +3,7 @@
3
3
  require 'i18n/tasks/scanners/file_scanner'
4
4
  require 'i18n/tasks/scanners/relative_keys'
5
5
  require 'i18n/tasks/scanners/ruby_ast_call_finder'
6
+ require 'i18n/tasks/scanners/ast_matchers/default_i18n_subject_matcher'
6
7
  require 'i18n/tasks/scanners/ast_matchers/message_receivers_matcher'
7
8
  require 'i18n/tasks/scanners/ast_matchers/rails_model_matcher'
8
9
  require 'parser/current'
@@ -80,7 +81,7 @@ module I18n::Tasks::Scanners
80
81
  results = []
81
82
 
82
83
  # method_name is not available at this stage
83
- calls.each do |send_node, _method_name|
84
+ calls.each do |(send_node, _method_name)|
84
85
  @matchers.each do |matcher|
85
86
  result = matcher.convert_to_key_occurrences(
86
87
  send_node,
@@ -5,6 +5,11 @@ module I18n
5
5
  module SplitKey
6
6
  module_function
7
7
 
8
+ PARENTHESIS_PAIRS = %w({} [] () <>).freeze
9
+ START_KEYS = PARENTHESIS_PAIRS.to_set { |pair| pair[0] }.freeze
10
+ END_KEYS = PARENTHESIS_PAIRS.to_h { |pair| [pair[0], pair[1]] }.freeze
11
+ private_constant :PARENTHESIS_PAIRS, :START_KEYS, :END_KEYS
12
+
8
13
  # split a key by dots (.)
9
14
  # dots inside braces or parenthesis are not split on
10
15
  #
@@ -12,61 +17,39 @@ module I18n
12
17
  # split_key 'a.#{b.c}' # => ['a', '#{b.c}']
13
18
  # split_key 'a.b.c', 2 # => ['a', 'b.c']
14
19
  def split_key(key, max = Float::INFINITY)
15
- parts = []
16
- pos = 0
17
20
  return [key] if max == 1
18
21
 
19
- key_parts(key) do |part|
20
- parts << part
21
- pos += part.length + 1
22
- if parts.length + 1 >= max
23
- parts << key[pos..] unless pos == key.length
24
- break
22
+ parts = []
23
+ current_parenthesis_end_char = nil
24
+ part = ''
25
+ key.each_char.with_index do |char, index|
26
+ if current_parenthesis_end_char
27
+ part += char
28
+ current_parenthesis_end_char = nil if char == current_parenthesis_end_char
29
+ elsif START_KEYS.include?(char)
30
+ part += char
31
+ current_parenthesis_end_char = END_KEYS[char]
32
+ elsif char == '.'
33
+ parts << part
34
+ if parts.size + 1 == max
35
+ remaining = key[(index + 1)..]
36
+ parts << remaining unless remaining.empty?
37
+ return parts
38
+ end
39
+ part = ''
40
+ else
41
+ part += char
25
42
  end
26
43
  end
27
- parts
28
- end
29
-
30
- def last_key_part(key)
31
- last = nil
32
- key_parts(key) { |part| last = part }
33
- last
34
- end
35
44
 
36
- # yield each key part
37
- # dots inside braces or parenthesis are not split on
38
- def key_parts(key, &block)
39
- return enum_for(:key_parts, key) unless block
45
+ return parts if part.empty?
40
46
 
41
- nesting = PARENS
42
- counts = PARENS_ZEROS # dup'd later if key contains parenthesis
43
- delim = '.'
44
- from = to = 0
45
- key.each_char do |char|
46
- if char == delim && PARENS_ZEROS == counts
47
- block.yield key[from...to]
48
- from = to = (to + 1)
49
- else
50
- nest_i, nest_inc = nesting[char]
51
- if nest_i
52
- counts = counts.dup if counts.frozen?
53
- counts[nest_i] += nest_inc
54
- end
55
- to += 1
56
- end
57
- end
58
- block.yield(key[from...to]) if from < to && to <= key.length
59
- true
47
+ current_parenthesis_end_char ? parts.concat(part.split('.')) : parts << part
60
48
  end
61
49
 
62
- PARENS = %w({} [] ()).each_with_object({}) do |s, h|
63
- i = h.size / 2
64
- h[s[0].freeze] = [i, 1].freeze
65
- h[s[1].freeze] = [i, -1].freeze
66
- end.freeze
67
- PARENS_ZEROS = Array.new(PARENS.size, 0).freeze
68
- private_constant :PARENS
69
- private_constant :PARENS_ZEROS
50
+ def last_key_part(key)
51
+ split_key(key).last
52
+ end
70
53
  end
71
54
  end
72
55
  end
@@ -70,6 +70,9 @@ module I18n::Tasks
70
70
  when Array
71
71
  # dump recursively
72
72
  value.map { |v| dump_value(v, opts) }
73
+ when Hash
74
+ # dump recursively
75
+ value.values.map { |v| dump_value(v, opts) }
73
76
  when String
74
77
  value = CGI.escapeHTML(value) if opts[:html_escape]
75
78
  replace_interpolations value unless value.empty?
@@ -85,6 +88,9 @@ module I18n::Tasks
85
88
  when Array
86
89
  # implode array
87
90
  untranslated.map { |from| parse_value(from, each_translated, opts) }
91
+ when Hash
92
+ # implode hash
93
+ untranslated.transform_values { |value| parse_value(value, each_translated, opts) }
88
94
  when String
89
95
  if untranslated.empty?
90
96
  untranslated
@@ -24,7 +24,12 @@ module I18n::Tasks::Translators
24
24
  def translate_values(list, from:, to:, **options)
25
25
  results = []
26
26
  list.each_slice(BATCH_SIZE) do |parts|
27
- res = DeepL.translate(parts, to_deepl_source_locale(from), to_deepl_target_locale(to), options)
27
+ res = DeepL.translate(
28
+ parts,
29
+ to_deepl_source_locale(from),
30
+ to_deepl_target_locale(to),
31
+ options_with_glossary(options, from, to)
32
+ )
28
33
  if res.is_a?(DeepL::Resources::Text)
29
34
  results << res.text
30
35
  else
@@ -100,5 +105,26 @@ module I18n::Tasks::Translators
100
105
  config.version = version unless version.blank?
101
106
  end
102
107
  end
108
+
109
+ def options_with_glossary(options, from, to)
110
+ glossary = find_glossary(from, to)
111
+ glossary ? { glossary_id: glossary.id }.merge(options) : options
112
+ end
113
+
114
+ def all_ready_glossaries
115
+ @all_ready_glossaries ||= DeepL.glossaries.list
116
+ end
117
+
118
+ def find_glossary(from, to)
119
+ config_glossary_ids = @i18n_tasks.translation_config[:deepl_glossary_ids]
120
+ return unless config_glossary_ids
121
+
122
+ all_ready_glossaries.find do |glossary|
123
+ glossary.ready \
124
+ && glossary.source_lang == from \
125
+ && glossary.target_lang == to \
126
+ && config_glossary_ids.include?(glossary.id)
127
+ end
128
+ end
103
129
  end
104
130
  end
@@ -4,6 +4,7 @@ require 'i18n/tasks/translators/base_translator'
4
4
 
5
5
  module I18n::Tasks::Translators
6
6
  class GoogleTranslator < BaseTranslator
7
+ NEWLINE_PLACEHOLDER = '<br id=i18n />'
7
8
  def initialize(*)
8
9
  begin
9
10
  require 'easy_translate'
@@ -16,14 +17,21 @@ module I18n::Tasks::Translators
16
17
  protected
17
18
 
18
19
  def translate_values(list, **options)
19
- EasyTranslate.translate(list, options)
20
+ restore_newlines(
21
+ EasyTranslate.translate(
22
+ replace_newlines_with_placeholder(list, options[:html]),
23
+ options,
24
+ format: :text
25
+ ),
26
+ options[:html]
27
+ )
20
28
  end
21
29
 
22
30
  def options_for_translate_values(from:, to:, **options)
23
31
  options.merge(
24
32
  api_key: api_key,
25
- from: to_google_translate_compatible_locale(from),
26
- to: to_google_translate_compatible_locale(to)
33
+ from: from,
34
+ to: to
27
35
  )
28
36
  end
29
37
 
@@ -41,15 +49,6 @@ module I18n::Tasks::Translators
41
49
 
42
50
  private
43
51
 
44
- SUPPORTED_LOCALES_WITH_REGION = %w[zh-CN zh-TW].freeze
45
-
46
- # Convert 'es-ES' to 'es'
47
- def to_google_translate_compatible_locale(locale)
48
- return locale unless locale.include?('-') && !SUPPORTED_LOCALES_WITH_REGION.include?(locale)
49
-
50
- locale.split('-', 2).first
51
- end
52
-
53
52
  def api_key
54
53
  @api_key ||= begin
55
54
  key = @i18n_tasks.translation_config[:google_translate_api_key]
@@ -65,5 +64,21 @@ module I18n::Tasks::Translators
65
64
  key
66
65
  end
67
66
  end
67
+
68
+ def replace_newlines_with_placeholder(list, html)
69
+ return list unless html
70
+
71
+ list.map do |value|
72
+ value.gsub("\n", NEWLINE_PLACEHOLDER)
73
+ end
74
+ end
75
+
76
+ def restore_newlines(translations, html)
77
+ return translations unless html
78
+
79
+ translations.map do |translation|
80
+ translation.gsub("#{NEWLINE_PLACEHOLDER} ", "\n")
81
+ end
82
+ end
68
83
  end
69
84
  end
@@ -1,11 +1,24 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  require 'i18n/tasks/translators/base_translator'
4
+ require 'active_support/core_ext/string/filters'
4
5
 
5
6
  module I18n::Tasks::Translators
6
7
  class OpenAiTranslator < BaseTranslator
7
8
  # max allowed texts per request
8
9
  BATCH_SIZE = 50
10
+ DEFAULT_SYSTEM_PROMPT = <<~PROMPT.squish
11
+ You are a professional translator that translates content from the %{from} locale
12
+ to the %{to} locale in an i18n locale array.
13
+
14
+ The array has a structured format and contains multiple strings. Your task is to translate
15
+ each of these strings and create a new array with the translated strings.
16
+
17
+ HTML markups (enclosed in < and > characters) must not be changed under any circumstance.
18
+ Variables (starting with %%{ and ending with }) must not be changed under any circumstance.
19
+
20
+ Keep in mind the context of all the strings for a more accurate translation.
21
+ PROMPT
9
22
 
10
23
  def initialize(*)
11
24
  begin
@@ -50,6 +63,14 @@ module I18n::Tasks::Translators
50
63
  end
51
64
  end
52
65
 
66
+ def model
67
+ @model ||= @i18n_tasks.translation_config[:openai_model].presence || 'gpt-3.5-turbo'
68
+ end
69
+
70
+ def system_prompt
71
+ @system_prompt ||= @i18n_tasks.translation_config[:openai_system_prompt].presence || DEFAULT_SYSTEM_PROMPT
72
+ end
73
+
53
74
  def translate_values(list, from:, to:)
54
75
  results = []
55
76
 
@@ -62,14 +83,11 @@ module I18n::Tasks::Translators
62
83
  results.flatten
63
84
  end
64
85
 
65
- def translate(values, from, to) # rubocop:disable Metrics/MethodLength
86
+ def translate(values, from, to)
66
87
  messages = [
67
88
  {
68
89
  role: 'system',
69
- content: "You are a helpful assistant that translates content from the #{from} to #{to} locale in an i18n
70
- locale array. The array has a structured format and contains multiple strings. Your task is to translate
71
- each of these strings and create a new array with the translated strings. Keep in mind the context of all
72
- the strings for a more accurate translation.\n"
90
+ content: format(system_prompt, from: from, to: to)
73
91
  },
74
92
  {
75
93
  role: 'user',
@@ -83,9 +101,9 @@ module I18n::Tasks::Translators
83
101
 
84
102
  response = translator.chat(
85
103
  parameters: {
86
- model: 'gpt-3.5-turbo',
104
+ model: model,
87
105
  messages: messages,
88
- temperature: 0.7
106
+ temperature: 0.0
89
107
  }
90
108
  )
91
109
 
@@ -21,8 +21,8 @@ module I18n::Tasks
21
21
  relative_roots: %w[app/controllers app/helpers app/mailers app/presenters app/views].freeze,
22
22
  scanners: [
23
23
  ['::I18n::Tasks::Scanners::RubyAstScanner', { only: %w[*.rb] }],
24
- ['::I18n::Tasks::Scanners::ErbAstScanner', { only: %w[*.erb] }],
25
- ['::I18n::Tasks::Scanners::PatternWithScopeScanner', { exclude: %w[*.erb *.rb] }]
24
+ ['::I18n::Tasks::Scanners::ErbAstScanner', { only: %w[*.html.erb] }],
25
+ ['::I18n::Tasks::Scanners::PatternWithScopeScanner', { exclude: %w[*.html.erb *.rb] }]
26
26
  ],
27
27
  ast_matchers: [],
28
28
  strict: true
@@ -2,6 +2,6 @@
2
2
 
3
3
  module I18n
4
4
  module Tasks
5
- VERSION = '1.0.13'
5
+ VERSION = '1.0.14'
6
6
  end
7
7
  end
@@ -13,7 +13,7 @@ data:
13
13
  ## Provide a custom adapter:
14
14
  # adapter: I18n::Tasks::Data::FileSystem
15
15
 
16
- # Locale files or `Find.find` patterns where translations are read from:
16
+ # Locale files or `Dir.glob` patterns where translations are read from:
17
17
  read:
18
18
  ## Default:
19
19
  # - config/locales/%{locale}.yml
@@ -92,9 +92,13 @@ search:
92
92
  ## - RailsModelMatcher
93
93
  ## Matches ActiveRecord translations like
94
94
  ## User.human_attribute_name(:email) and User.model_name.human
95
+ ## - DefaultI18nSubjectMatcher
96
+ ## Matches ActionMailer's default_i18n_subject method
95
97
  ##
96
98
  ## To implement your own, please see `I18n::Tasks::Scanners::AstMatchers::BaseMatcher`.
97
- # <%# I18n::Tasks.add_ast_matcher('I18n::Tasks::Scanners::AstMatchers::RailsModelMatcher') %>
99
+ # ast_matchers:
100
+ # - 'I18n::Tasks::Scanners::AstMatchers::RailsModelMatcher'
101
+ # - 'I18n::Tasks::Scanners::AstMatchers::DefaultI18nSubjectMatcher'
98
102
 
99
103
  ## Multiple scanners can be used. Their results are merged.
100
104
  ## The options specified above are passed down to each scanner. Per-scanner options can be specified as well.
@@ -110,9 +114,28 @@ search:
110
114
  # deepl_api_key: "48E92789-57A3-466A-9959-1A1A1A1A1A1A"
111
115
  # # deepl_host: "https://api.deepl.com"
112
116
  # # deepl_version: "v2"
117
+ # # deepl_glossary_ids:
118
+ # # - f28106eb-0e06-489e-82c6-8215d6f95089
119
+ # # - 2c6415be-1852-4f54-9e1b-d800463496b4
113
120
  # # add additional options to the DeepL.translate call: https://www.deepl.com/docs-api/translate-text/translate-text/
114
121
  # deepl_options:
115
122
  # formality: prefer_less
123
+ # # OpenAI
124
+ # openai_api_key: "sk-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
125
+ # # openai_model: "gpt-3.5-turbo" # see https://platform.openai.com/docs/models
126
+ # # may contain `%{from}` and `%{to}`, which will be replaced by source and target locale codes, respectively (using `Kernel.format`)
127
+ # # openai_system_prompt: >-
128
+ # # You are a professional translator that translates content from the %{from} locale
129
+ # # to the %{to} locale in an i18n locale array.
130
+ # #
131
+ # # The array has a structured format and contains multiple strings. Your task is to translate
132
+ # # each of these strings and create a new array with the translated strings.
133
+ # #
134
+ # # HTML markups (enclosed in < and > characters) must not be changed under any circumstance.
135
+ # # Variables (starting with %%{ and ending with }) must not be changed under any circumstance.
136
+ # #
137
+ # # Keep in mind the context of all the strings for a more accurate translation.
138
+
116
139
  ## Do not consider these keys missing:
117
140
  # ignore_missing:
118
141
  # - 'errors.messages.{accepted,blank,invalid,too_short,too_long}'
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: i18n-tasks
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.0.13
4
+ version: 1.0.14
5
5
  platform: ruby
6
6
  authors:
7
7
  - glebm
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2023-10-14 00:00:00.000000000 Z
11
+ date: 2024-05-10 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: activesupport
@@ -38,26 +38,6 @@ dependencies:
38
38
  - - ">="
39
39
  - !ruby/object:Gem::Version
40
40
  version: 2.1.0
41
- - !ruby/object:Gem::Dependency
42
- name: better_html
43
- requirement: !ruby/object:Gem::Requirement
44
- requirements:
45
- - - ">="
46
- - !ruby/object:Gem::Version
47
- version: '1.0'
48
- - - "<"
49
- - !ruby/object:Gem::Version
50
- version: '3.0'
51
- type: :runtime
52
- prerelease: false
53
- version_requirements: !ruby/object:Gem::Requirement
54
- requirements:
55
- - - ">="
56
- - !ruby/object:Gem::Version
57
- version: '1.0'
58
- - - "<"
59
- - !ruby/object:Gem::Version
60
- version: '3.0'
61
41
  - !ruby/object:Gem::Dependency
62
42
  name: erubi
63
43
  requirement: !ruby/object:Gem::Requirement
@@ -388,6 +368,7 @@ files:
388
368
  - lib/i18n/tasks/data/file_system.rb
389
369
  - lib/i18n/tasks/data/file_system_base.rb
390
370
  - lib/i18n/tasks/data/router/conservative_router.rb
371
+ - lib/i18n/tasks/data/router/isolating_router.rb
391
372
  - lib/i18n/tasks/data/router/pattern_router.rb
392
373
  - lib/i18n/tasks/data/tree/node.rb
393
374
  - lib/i18n/tasks/data/tree/nodes.rb
@@ -406,9 +387,9 @@ files:
406
387
  - lib/i18n/tasks/reports/base.rb
407
388
  - lib/i18n/tasks/reports/terminal.rb
408
389
  - lib/i18n/tasks/scanners/ast_matchers/base_matcher.rb
390
+ - lib/i18n/tasks/scanners/ast_matchers/default_i18n_subject_matcher.rb
409
391
  - lib/i18n/tasks/scanners/ast_matchers/message_receivers_matcher.rb
410
392
  - lib/i18n/tasks/scanners/ast_matchers/rails_model_matcher.rb
411
- - lib/i18n/tasks/scanners/erb_ast_processor.rb
412
393
  - lib/i18n/tasks/scanners/erb_ast_scanner.rb
413
394
  - lib/i18n/tasks/scanners/file_scanner.rb
414
395
  - lib/i18n/tasks/scanners/files/caching_file_finder.rb
@@ -474,7 +455,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
474
455
  - !ruby/object:Gem::Version
475
456
  version: '0'
476
457
  requirements: []
477
- rubygems_version: 3.2.3
458
+ rubygems_version: 3.5.3
478
459
  signing_key:
479
460
  specification_version: 4
480
461
  summary: Manage localization and translation with the awesome power of static analysis
@@ -1,74 +0,0 @@
1
- # frozen_string_literal: true
2
-
3
- require 'ast'
4
- require 'set'
5
- require 'i18n/tasks/scanners/local_ruby_parser'
6
-
7
- module I18n::Tasks::Scanners
8
- class ErbAstProcessor
9
- include AST::Processor::Mixin
10
- def initialize
11
- super()
12
- @ruby_parser = LocalRubyParser.new(ignore_blocks: true)
13
- @comments = []
14
- end
15
-
16
- def process_and_extract_comments(ast)
17
- result = process(ast)
18
- [result, @comments]
19
- end
20
-
21
- def on_code(node)
22
- parsed, comments = @ruby_parser.parse(
23
- node.children[0],
24
- location: node.location
25
- )
26
- @comments.concat(comments)
27
-
28
- unless parsed.nil?
29
- parsed = parsed.updated(
30
- nil,
31
- parsed.children.map { |child| node?(child) ? process(child) : child }
32
- )
33
- node = node.updated(:send, parsed)
34
- end
35
- node
36
- end
37
-
38
- # @param node [::Parser::AST::Node]
39
- # @return [::Parser::AST::Node]
40
- def handler_missing(node)
41
- node = handle_comment(node)
42
- return if node.nil?
43
-
44
- node.updated(
45
- nil,
46
- node.children.map { |child| node?(child) ? process(child) : child }
47
- )
48
- end
49
-
50
- private
51
-
52
- # Convert ERB-comments to ::Parser::Source::Comment and skip processing node
53
- #
54
- # @param node Parser::AST::Node Potential comment node
55
- # @return Parser::AST::Node or nil
56
- def handle_comment(node)
57
- if node.type == :erb && node.children.size == 4 &&
58
- node.children[0]&.type == :indicator && node.children[0].children[0] == '#' &&
59
- node.children[2]&.type == :code
60
-
61
- # Do not continue parsing this node
62
- comment = node.children[2]
63
- @comments << ::Parser::Source::Comment.new(comment.location.expression)
64
- return
65
- end
66
-
67
- node
68
- end
69
-
70
- def node?(node)
71
- node.is_a?(::Parser::AST::Node)
72
- end
73
- end
74
- end