i18n-tasks 1.0.14 → 1.0.15

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: bd61ceb254dc1c44768bd40672b3f31635c9d42fb237950cbdfe64fef642a850
-  data.tar.gz: ae6981dd739636ddb5f1bae1ca9e5cf05ba991fd1345a5488e1cfa8561833877
+  metadata.gz: 6c2d36ee3f03672492a67bc2b98344a65051392dfaf0891aee1cbe1c01126884
+  data.tar.gz: 1e8ff7f7b02d804be0c4d077424cec4dfbb5881b3bcada145b1c2878e3913e2b
 SHA512:
-  metadata.gz: 1312910139411cc4695bea0f0de8624d49958efac33c8c7b7bd99fb3f351d408d0a6980789ec81a43452bc29a15292f6de9719ace0e3b902866d63c02f5cab79
-  data.tar.gz: e782f1bc1f1ad68cc33583d014c879b3d121ba2e0990db8cc109f864aeb0e98f0a97030300294e31e9041899b131b2b677fe6917f705d09bda6bf07aed588b42
+  metadata.gz: 1955ea4adbf216f1e4014ea4b5f3873d45a14304a7c5d36b1692f5bd6924f358dd2bd90154d3a857678b50860ee1c248a2d914eef98af54062f8f03aced8a0ce
+  data.tar.gz: 9a3e08a0a8ab27242701ea48dc68be4635e7f553d647e7f27d1449be51a014f0b0fa8d35bccd67ff2402ad9d25400f4463d702bac9cf5e285786211d4c451635

data/README.md

@@ -24,7 +24,7 @@ i18n-tasks can be used with any project using the ruby [i18n gem][i18n-gem] (def
 Add i18n-tasks to the Gemfile:
 
 ```ruby
-gem 'i18n-tasks', '~> 1.0.14', group: :development
+gem 'i18n-tasks', '~> 1.0.15', group: :development
 ```
 
 Copy the default [configuration file](#configuration):
@@ -84,49 +84,23 @@ Usage: i18n-tasks add-missing [options] [locale ...]
     -h, --help     Display this help message.
 ```
 
-### Google Translate missing keys
+### Translate Missing Keys
 
-Translate missing values with Google Translate ([more below on the API key](#google-translation-config)).
+Translate missing keys using a backend service of your choice.
 
 ```console
 $ i18n-tasks translate-missing
 
-# accepts from and locales options:
-$ i18n-tasks translate-missing --from=base es fr
+# accepts backend, from and locales options
+$ i18n-tasks translate-missing --from=base es fr --backend=google
 ```
 
-### DeepL Pro Translate missing keys
-
-Translate missing values with DeepL Pro Translate ([more below on the API key](#deepl-translation-config)).
-
-```console
-$ i18n-tasks translate-missing --backend=deepl
-
-# accepts from and locales options:
-$ i18n-tasks translate-missing --backend=deepl --from=en fr nl
-```
-
-### Yandex Translate missing keys
-
-Translate missing values with Yandex Translate ([more below on the API key](#yandex-translation-config)).
-
-```console
-$ i18n-tasks translate-missing --backend=yandex
-
-# accepts from and locales options:
-$ i18n-tasks translate-missing --from=en es fr
-```
-
-### OpenAI Translate missing keys
-
-Translate missing values with OpenAI ([more below on the API key](#openai-translation-config)).
-
-```console
-$ i18n-tasks translate-missing --backend=openai
-
-# accepts from and locales options:
-$ i18n-tasks translate-missing --from=en es fr
-```
+Available backends:
+- `google` - [Google Translate](#google-translation-config)
+- `deepl` - [DeepL Pro](#deepl-translation-config)
+- `yandex` - [Yandex Translate](#yandex-translation-config)
+- `openai` - [OpenAI](#openai-translation-config)
+- `watsonx` - [watsonx](#watsonx-translation-config)
 
 ### Find usages
 
@@ -435,6 +409,7 @@ Put the key in `GOOGLE_TRANSLATE_API_KEY` environment variable or in the config
 ```yaml
 # config/i18n-tasks.yml
 translation:
+  backend: google
   google_translate_api_key: <Google Translate API key>
 ```
 
@@ -447,11 +422,12 @@ GOOGLE_TRANSLATE_API_KEY=<Google Translate API key>
 <a name="deepl-translation-config"></a>
 ### DeepL Pro Translate
 
-`i18n-tasks translate-missing` requires a DeepL Pro API key, get it at [DeepL](https://www.deepl.com/pro).
+`i18n-tasks translate-missing` requires a DeepL Pro API key, get it at [DeepL](https://www.deepl.com/pro). You can specify alias locales if you only use the simple locales internally.
 
 ```yaml
 # config/i18n-tasks.yml
 translation:
+  backend: deepl
   deepl_api_key: <DeepL Pro API key>
   deepl_host: <optional>
   deepl_version: <optional>
@@ -460,6 +436,9 @@ translation:
     - 2c6415be-1852-4f54-9e1b-d800463496b4
   deepl_options:
     formality: prefer_less
+  deepl_locale_aliases:
+    en: en-us
+    pt: pt-br
 ```
 
 or via environment variables:
@@ -478,6 +457,7 @@ DEEPL_VERSION=<optional>
 ```yaml
 # config/i18n-tasks.yml
 translation:
+  backend: yandex
   yandex_api_key: <Yandex API key>
 ```
 
@@ -495,6 +475,7 @@ YANDEX_API_KEY=<Yandex API key>
 ```yaml
 # config/i18n-tasks.yml
 translation:
+  backend: openai
   openai_api_key: <OpenAI API key>
   openai_model: <optional>
 ```
@@ -506,6 +487,52 @@ OPENAI_API_KEY=<OpenAI API key>
 OPENAI_MODEL=<optional>
 ```
 
+<a name="watsonx-translation-config"></a>
+### watsonx Translate
+
+`i18n-tasks translate-missing` requires a watsonx project and api key, get it at [IBM watsonx](https://www.ibm.com/watsonx/).
+
+```yaml
+# config/i18n-tasks.yml
+translation:
+  backend: watsonx
+  watsonx_api_key: <watsonx API key>
+  watsonx_project_id: <watsonx project id>
+  watsonx_model: <optional>
+```
+
+or via environment variable:
+
+```bash
+WATSONX_API_KEY=<watsonx API key>
+WATSONX_PROJECT_ID=<watsonx project id>
+WATSONX_MODEL=<optional>
+```
+
+### Contextual Rails Parser
+
+There is an experimental feature to parse Rails with more context. `i18n-tasks` will support:
+- Translations called in `before_actions`
+- Translations called in nested methods
+- `Model.human_attribute_name` calls
+- `Model.model_name.human` calls
+
+Enabled it by adding the scanner in your `config/i18n-tasks.yml`:
+
+```ruby
+<% I18n::Tasks.add_scanner( 
+  'I18n::Tasks::Scanners::PrismScanner',
+  only: %w(*.rb)
+) %>
+```
+
+To only enable Ruby-scanning and not any Rails support, please add config under the `search` section:
+
+```yaml
+search:
+  prism_visitor: "ruby" # default "rails"
+```
+
 ## Interactive console
 
 `i18n-tasks irb` starts an IRB session in i18n-tasks context. Type `guide` for more information.

data/config/locales/en.yml

@@ -26,7 +26,7 @@ en:
           strict: >-
             Avoid inferring dynamic key usages such as t("cats.#{cat}.name"). Takes precedence over
             the config setting if set.
-          translation_backend: Translation backend (google or deepl)
+          translation_backend: Translation backend [google, deepl, yandex, openai])
           value: >-
             Value. Interpolates: %%{value}, %%{human_key}, %%{key}, %%{default}, %%{value_or_human_key},
             %%{value_or_default_or_human_key}
@@ -69,6 +69,7 @@ en:
       enum_opt:
         invalid: "%{invalid} is not one of: %{valid}."
       errors:
+        invalid_backend: 'Invalid backend: %{invalid}. Must be one of %{valid}.'
         invalid_format: 'invalid format: %{invalid}. valid: %{valid}.'
         invalid_locale: 'invalid locale: %{invalid}'
         invalid_missing_type:
@@ -129,6 +130,15 @@ en:
       none: Every translation is in use.
     usages:
       none: No key usages found.
+    watsonx_translate:
+      errors:
+        no_api_key: >-
+          Set watsonx API key via WATSONX_API_KEY environment variable or translation.watsonx_api_key
+          in config/i18n-tasks.yml. Get the key at https://www.ibm.com/products/watsonx-ai.
+        no_project_id: >-
+          Set watsonx Project ID via WATSONX_PROJECT_ID environment variable or translation.watsonx_api_key
+          in config/i18n-tasks.yml. Get the key at https://www.ibm.com/products/watsonx-ai.
+        no_results: watsonx returned no results.
     yandex_translate:
       errors:
         no_api_key: >-

data/config/locales/ru.yml

@@ -23,7 +23,7 @@ ru:
           out_format: 'Формат вывода: %{valid_text}.'
           pattern_router: 'Использовать pattern_router: ключи распределятся по файлам согласно data.write'
           strict: Не угадывать динамические использования ключей, например `t("category.#{category.key}")`
-          translation_backend: Движок перевода (google или deepl)
+          translation_backend: Движок перевода [google, deepl, yandex, openai]
           value: >-
             Значение, интерполируется с %%{value}, %%{human_key}, %%{key}, %%{default}, %%{value_or_human_key},
             %%{value_or_default_or_human_key}
@@ -66,6 +66,7 @@ ru:
       enum_opt:
         invalid: "%{invalid} не является одним из: %{valid}."
       errors:
+        invalid_backend: 'Недопустимый источник данных: %{invalid}. Должен быть одним из %{valid}.'
         invalid_format: 'Неизвестный формат %{invalid}. Форматы: %{valid}.'
         invalid_locale: Неверный язык %{invalid}
         invalid_missing_type:
@@ -130,6 +131,15 @@ ru:
       none: Все переводы используются.
     usages:
       none: Не найдено использований.
+    watsonx_translate:
+      errors:
+        no_api_key: >-
+          Установите ключ API watsonx через переменную среды WATSONX_API_KEY или translation.watsonx_api_key
+          в config/i18n-tasks.yml. Получите ключ на https://www.ibm.com/products/watsonx-ai.
+        no_project_id: >-
+          Установите идентификатор проекта watsonx через переменную среды WATSONX_PROJECT_ID или translation.watsonx_api_key
+          в config/i18n-tasks.yml. Получите ключ на https://www.ibm.com/products/watsonx-ai.
+        no_results: watsonx не вернул результатов.
     yandex_translate:
       errors:
         no_api_key: |-

data/i18n-tasks.gemspec

@@ -27,8 +27,10 @@ Gem::Specification.new do |s|
   TEXT
   s.homepage = 'https://github.com/glebm/i18n-tasks'
   s.metadata = {
+    'changelog_uri' => 'https://github.com/glebm/i18n-tasks/blob/main/CHANGES.md',
     'issue_tracker' => 'https://github.com/glebm/i18n-tasks',
-    'rubygems_mfa_required' => 'true'
+    'rubygems_mfa_required' => 'true',
+    'source_code_uri' => 'https://github.com/glebm/i18n-tasks'
   }
   s.required_ruby_version = '>= 2.6', '< 4.0'
 
@@ -45,6 +47,7 @@ Gem::Specification.new do |s|
   s.add_dependency 'parser', '>= 3.2.2.1'
   s.add_dependency 'rails-i18n'
   s.add_dependency 'rainbow', '>= 2.2.2', '< 4.0'
+  s.add_dependency 'ruby-progressbar', '~> 1.8', '>= 1.8.1'
   s.add_dependency 'terminal-table', '>= 1.5.1'
   s.add_development_dependency 'bundler', '~> 2.0', '>= 2.0.1'
   s.add_development_dependency 'overcommit', '~> 0.58.0'

data/lib/i18n/tasks/command/commands/missing.rb

@@ -48,7 +48,9 @@ module I18n::Tasks
             pattern_re = i18n.compile_key_pattern(opt[:pattern])
             missing.select_keys! { |full_key, _node| full_key =~ pattern_re }
           end
-          translated = i18n.translate_forest missing, from: opt[:from], backend: opt[:backend].to_sym
+
+          backend = opt[:backend].presence || i18n.translation_config[:backend]
+          translated = i18n.translate_forest missing, from: opt[:from], backend: backend.to_sym
           i18n.data.merge! translated
           log_stderr t('i18n_tasks.translate_missing.translated', count: translated.leaves.count)
           print_forest translated, opt

data/lib/i18n/tasks/command/option_parsers/enum.rb

@@ -9,17 +9,18 @@ module I18n::Tasks
             I18n.t('i18n_tasks.cmd.enum_opt.invalid', invalid: invalid, valid: valid * ', ')
           end
 
-          def initialize(valid, error_message = DEFAULT_ERROR)
+          def initialize(valid, error_message = DEFAULT_ERROR, allow_blank: false)
             @valid         = valid.map(&:to_s)
             @error_message = error_message
+            @allow_blank   = allow_blank
           end
 
           def call(value, *)
-            return @valid.first unless value.present?
+            return @valid.first if value.blank? && !@allow_blank
 
             if @valid.include?(value)
               value
-            else
+            elsif value.present? || !@allow_blank
               fail CommandError, @error_message.call(value, @valid)
             end
           end

data/lib/i18n/tasks/command/options/locales.rb

@@ -1,6 +1,7 @@
 # frozen_string_literal: true
 
 require 'i18n/tasks/command/option_parsers/locale'
+require 'i18n/tasks/command/option_parsers/enum'
 
 module I18n::Tasks
   module Command
@@ -31,13 +32,21 @@ module I18n::Tasks
             parser: OptionParsers::Locale::Parser,
             default: 'base'
 
-        TRANSLATION_BACKENDS = %w[google deepl].freeze
+        TRANSLATION_BACKENDS = %w[google deepl yandex openai watsonx].freeze
         arg :translation_backend,
             '-b',
             '--backend BACKEND',
             t('i18n_tasks.cmd.args.desc.translation_backend'),
-            parser: OptionParsers::Locale::Parser,
-            default: TRANSLATION_BACKENDS[0]
+            parser:
+              OptionParsers::Enum::Parser.new(
+                TRANSLATION_BACKENDS,
+                proc do |value, valid|
+                  if value.present?
+                    I18n.t('i18n_tasks.cmd.errors.invalid_backend', invalid: value&.strip, valid: valid * ', ')
+                  end
+                end,
+                allow_blank: true
+              )
       end
     end
   end

data/lib/i18n/tasks/configuration.rb

@@ -5,7 +5,8 @@ module I18n::Tasks::Configuration # rubocop:disable Metrics/ModuleLength
     base_locale: 'en',
     internal_locale: 'en',
     search: ::I18n::Tasks::UsedKeys::SEARCH_DEFAULTS,
-    data: ::I18n::Tasks::Data::DATA_DEFAULTS
+    data: ::I18n::Tasks::Data::DATA_DEFAULTS,
+    translation_backend: :google
   }.freeze
 
   # i18n-tasks config (defaults + config/i18n-tasks.yml)
@@ -59,15 +60,19 @@ module I18n::Tasks::Configuration # rubocop:disable Metrics/ModuleLength
 
   # translation config
   # @return [Hash{String => String,Hash,Array}]
-  def translation_config
+  def translation_config # rubocop:disable Metrics/AbcSize, Metrics/CyclomaticComplexity, Metrics/PerceivedComplexity
     @config_sections[:translation] ||= begin
       conf = (config[:translation] || {}).with_indifferent_access
+      conf[:backend] ||= DEFAULTS[:translation_backend]
       conf[:google_translate_api_key] = ENV['GOOGLE_TRANSLATE_API_KEY'] if ENV.key?('GOOGLE_TRANSLATE_API_KEY')
       conf[:deepl_api_key] = ENV['DEEPL_AUTH_KEY'] if ENV.key?('DEEPL_AUTH_KEY')
       conf[:deepl_host] = ENV['DEEPL_HOST'] if ENV.key?('DEEPL_HOST')
       conf[:deepl_version] = ENV['DEEPL_VERSION'] if ENV.key?('DEEPL_VERSION')
       conf[:openai_api_key] = ENV['OPENAI_API_KEY'] if ENV.key?('OPENAI_API_KEY')
       conf[:openai_model] = ENV['OPENAI_MODEL'] if ENV.key?('OPENAI_MODEL')
+      conf[:watsonx_api_key] = ENV['WATSONX_API_KEY'] if ENV.key?('WATSONX_API_KEY')
+      conf[:watsonx_project_id] = ENV['WATSONX_PROJECT_ID'] if ENV.key?('WATSONX_PROJECT_ID')
+      conf[:watsonx_model] = ENV['WATSONX_MODEL'] if ENV.key?('WATSONX_MODEL')
       conf[:yandex_api_key] = ENV['YANDEX_API_KEY'] if ENV.key?('YANDEX_API_KEY')
       conf
     end

data/lib/i18n/tasks/scanners/ast_matchers/rails_model_matcher.rb

@@ -16,7 +16,7 @@ module I18n::Tasks::Scanners::AstMatchers
       receiver = children[0]
       method_name = children[1]
 
-      return unless method_name == :human_attribute_name && receiver.type == :const
+      return unless method_name == :human_attribute_name && receiver&.type == :const
 
       value = children[2]
 

data/lib/i18n/tasks/scanners/local_ruby_parser.rb

@@ -1,6 +1,6 @@
 # frozen_string_literal: true
 
-require 'parser/current'
+require 'i18n/tasks/scanners/ruby_parser_factory'
 
 module I18n::Tasks::Scanners
   class LocalRubyParser
@@ -9,7 +9,7 @@ module I18n::Tasks::Scanners
     BLOCK_EXPR = /\s*((\s+|\))do|\{)(\s*\|[^|]*\|)?\s*\Z/.freeze
 
     def initialize(ignore_blocks: false)
-      @parser = ::Parser::CurrentRuby.new
+      @parser = RubyParserFactory.create_parser
       @ignore_blocks = ignore_blocks
     end
 

data/lib/i18n/tasks/scanners/pattern_scanner.rb

@@ -12,7 +12,7 @@ module I18n::Tasks::Scanners
     include OccurrenceFromPosition
     include RubyKeyLiterals
 
-    TRANSLATE_CALL_RE = /(?<=^|[^\p{L}'\-.]|[^\p{L}'-]I18n\.|I18n\.)t(?:!|ranslate!?)?/.freeze
+    TRANSLATE_CALL_RE = /(?<=^|[^\p{L}_'\-.]|[^\p{L}'-]I18n\.|I18n\.)t(?:!|ranslate!?)?/.freeze
     IGNORE_LINES = {
       'coffee' => /^\s*#(?!\si18n-tasks-use)/,
       'erb' => /^\s*<%\s*#(?!\si18n-tasks-use)/,

data/lib/i18n/tasks/scanners/prism_scanner.rb

@@ -0,0 +1,83 @@
+# frozen_string_literal: true
+
+require_relative 'file_scanner'
+require_relative 'ruby_ast_scanner'
+
+module I18n::Tasks::Scanners
+  class PrismScanner < FileScanner
+    MAGIC_COMMENT_SKIP_PRISM = 'i18n-tasks-skip-prism'
+
+    def initialize(**args)
+      unless VISITOR
+        warn(
+          'Please make sure `prism` is available to use this feature. Fallback to Ruby AST Scanner.'
+        )
+      end
+      super
+
+      @fallback = RubyAstScanner.new(**args)
+    end
+
+    protected
+
+    # Extract all occurrences of translate calls from the file at the given path.
+    #
+    # @return [Array<[key, Results::KeyOccurrence]>] each occurrence found in the file
+    def scan_file(path)
+      return @fallback.send(:scan_file, path) if VISITOR.nil?
+
+      process_results(path, PARSER.parse_file(path))
+    rescue Exception => e # rubocop:disable Lint/RescueException
+      raise(
+        ::I18n::Tasks::CommandError.new(
+          e,
+          "Error scanning #{path}: #{e.message}"
+        )
+      )
+    end
+
+    # Need to have method that can be overridden to be able to test it
+    def process_results(path, parse_results)
+      parsed = parse_results.value
+      comments = parse_results.comments
+
+      return @fallback.send(:scan_file, path) if skip_prism_comment?(comments)
+
+      rails = if config[:prism_visitor].blank?
+                true
+              else
+                config[:prism_visitor] != 'ruby'
+              end
+
+      visitor = VISITOR.new(comments: comments, rails: rails)
+      parsed.accept(visitor)
+
+      occurrences = []
+      visitor.process.each do |translation_call|
+        result = translation_call.occurrences(path)
+        occurrences << result if result
+      end
+
+      occurrences
+    end
+
+    def skip_prism_comment?(comments)
+      comments.any? do |comment|
+        content =
+          comment.respond_to?(:slice) ? comment.slice : comment.location.slice
+        content.include?(MAGIC_COMMENT_SKIP_PRISM)
+      end
+    end
+
+    # This block handles adding a fallback if the `prism` gem is not available.
+    begin
+      require 'prism'
+      require_relative 'prism_scanners/visitor'
+      PARSER = Prism
+      VISITOR = I18n::Tasks::Scanners::PrismScanners::Visitor
+    rescue LoadError
+      PARSER = nil
+      VISITOR = nil
+    end
+  end
+end

data/lib/i18n/tasks/scanners/prism_scanners/arguments_visitor.rb

@@ -0,0 +1,41 @@
+# frozen_string_literal: true
+
+require 'prism/visitor'
+
+# This class is used to parse the arguments to e.g. a Prism::CallNode and return the values we need
+# for turning them into translations and occurrences.
+# Used in the PrismScanners::Visitor class.
+module I18n::Tasks::Scanners::PrismScanners
+  class ArgumentsVisitor < Prism::Visitor
+    def visit_keyword_hash_node(node)
+      node.child_nodes.each_with_object({}) do |child, hash|
+        hash[visit(child.key)] = visit(child.value)
+        hash
+      end
+    end
+
+    def visit_symbol_node(node)
+      node.value
+    end
+
+    def visit_string_node(node)
+      node.content
+    end
+
+    def visit_array_node(node)
+      node.child_nodes.map { |child| visit(child) }
+    end
+
+    def visit_arguments_node(node)
+      node.child_nodes.map { |child| visit(child) }
+    end
+
+    def visit_integer_node(node)
+      node.value
+    end
+
+    def visit_lambda_node(node)
+      node
+    end
+  end
+end