i18n-tasks 1.0.14 → 1.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: bd61ceb254dc1c44768bd40672b3f31635c9d42fb237950cbdfe64fef642a850
-  data.tar.gz: ae6981dd739636ddb5f1bae1ca9e5cf05ba991fd1345a5488e1cfa8561833877
+  metadata.gz: 3231dd8c4f3772a49ed9db6862f6d34c9fabed656a1ba3588f2a0d24be158d50
+  data.tar.gz: b5db9e81af6eff9256acf3b84bbbcea4a738c987f0485e04e127eba0236f01fd
 SHA512:
-  metadata.gz: 1312910139411cc4695bea0f0de8624d49958efac33c8c7b7bd99fb3f351d408d0a6980789ec81a43452bc29a15292f6de9719ace0e3b902866d63c02f5cab79
-  data.tar.gz: e782f1bc1f1ad68cc33583d014c879b3d121ba2e0990db8cc109f864aeb0e98f0a97030300294e31e9041899b131b2b677fe6917f705d09bda6bf07aed588b42
+  metadata.gz: 601af6d9cf98382b36cb7eebec4672c8918440f63899ef8e91cda798de0685971e331e352420c0f7d3c2f4a5dc056d80f7b0feb1b04e7aa847c69ebd273ff214
+  data.tar.gz: d8e0c5b7bb14a6ab061f60360e869f293b2c608fde5a02f7ebb90495f8c0d0adec69d4665818bd96b2fe6f3c590434aa94ad35462e2777bd9925ac3150aa4273

data/README.md

@@ -24,7 +24,7 @@ i18n-tasks can be used with any project using the ruby [i18n gem][i18n-gem] (def
 Add i18n-tasks to the Gemfile:
 
 ```ruby
-gem 'i18n-tasks', '~> 1.0.14', group: :development
+gem 'i18n-tasks', '~> 1.1.0', group: :development
 ```
 
 Copy the default [configuration file](#configuration):
@@ -84,49 +84,23 @@ Usage: i18n-tasks add-missing [options] [locale ...]
     -h, --help     Display this help message.
 ```
 
-### Google Translate missing keys
+### Translate Missing Keys
 
-Translate missing values with Google Translate ([more below on the API key](#google-translation-config)).
+Translate missing keys using a backend service of your choice.
 
 ```console
 $ i18n-tasks translate-missing
 
-# accepts from and locales options:
-$ i18n-tasks translate-missing --from=base es fr
+# accepts backend, from and locales options
+$ i18n-tasks translate-missing --from=base es fr --backend=google
 ```
 
-### DeepL Pro Translate missing keys
-
-Translate missing values with DeepL Pro Translate ([more below on the API key](#deepl-translation-config)).
-
-```console
-$ i18n-tasks translate-missing --backend=deepl
-
-# accepts from and locales options:
-$ i18n-tasks translate-missing --backend=deepl --from=en fr nl
-```
-
-### Yandex Translate missing keys
-
-Translate missing values with Yandex Translate ([more below on the API key](#yandex-translation-config)).
-
-```console
-$ i18n-tasks translate-missing --backend=yandex
-
-# accepts from and locales options:
-$ i18n-tasks translate-missing --from=en es fr
-```
-
-### OpenAI Translate missing keys
-
-Translate missing values with OpenAI ([more below on the API key](#openai-translation-config)).
-
-```console
-$ i18n-tasks translate-missing --backend=openai
-
-# accepts from and locales options:
-$ i18n-tasks translate-missing --from=en es fr
-```
+Available backends:
+- `google` - [Google Translate](#google-translation-config)
+- `deepl` - [DeepL Pro](#deepl-translation-config)
+- `yandex` - [Yandex Translate](#yandex-translation-config)
+- `openai` - [OpenAI](#openai-translation-config)
+- `watsonx` - [watsonx](#watsonx-translation-config)
 
 ### Find usages
 
@@ -253,6 +227,27 @@ See the full list of tasks with `i18n-tasks --help`.
 ✔ Reference keys (keys with `:symbol` values) are fully supported. These keys are copied as-is in
 `add/translate-missing`, and can be looked up by reference or value in `find`.
 
+#### Unexpected normalization
+`i18n-tasks` uses a yaml parser and emitter called `Psych` under the hood. `Psych` has it's own heuristic on when
+to use `|`, `>`, or `""` for multi-line strings. This can have some unexpected consequences, eg when normalizing:
+```yaml
+a: | 
+  Lorem ipsum dolor sit amet, consectetur
+  Lorem ipsum dolor sit amet, consectetur
+b: | 
+  Lorem ipsum dolor sit amet, consectetur 
+  Lorem ipsum dolor sit amet, consectetur 
+```
+we get the result
+```yaml
+a: |
+  Lorem ipsum dolor sit amet, consectetur
+  Lorem ipsum dolor sit amet, consectetur
+b: "Lorem ipsum dolor sit amet, consectetur \nLorem ipsum dolor sit amet, consectetur\n"
+```
+The only difference between `a` and `b` is that `b` has an extra trailing space in each line.
+This is an unfortunate side effect of `i18n-tasks` using `Psych`.
+
 #### `t()` keyword arguments
 
 ✔ `scope` keyword argument is fully supported by the AST scanner, and also by the Regexp scanner but only when it is the first argument.
@@ -392,6 +387,22 @@ truck.contents.description_body ⮕ truck.attributes.description.body
 If you store data somewhere but in the filesystem, e.g. in the database or mongodb, you can implement a custom adapter.
 If you have implemented a custom adapter please share it on [the wiki][wiki].
 
+#### Rails credentials
+
+If you use Rails credentials and want to load e.g. credentials for translation backends, convert your `i18n-tasks.yml`to `i18n-tasks.yml.erb` and add
+a `require "./config/application"` line to load Rails.
+
+```yaml
+# config/i18n-tasks.yml.erb
+<% require "./config/application" %>
+
+# ...
+
+translation:
+  backend: google
+  google_translate_api_key: <%= Rails.application.credentials.google_translate_api_key %>
+```
+
 ### Usage search
 
 i18n-tasks uses an AST scanner for `.rb` and `.html.erb` files, and a regexp scanner for all other files.
@@ -418,6 +429,24 @@ For more complex cases, you can implement a [custom scanner][custom-scanner-docs
 
 See the [config file][config] to find out more.
 
+### Environment Variables and Dotenv
+
+i18n-tasks supports loading environment variables from `.env` files using the [dotenv](https://github.com/bkeepers/dotenv) gem.
+This is particularly useful for storing translation API keys and other sensitive configuration.
+
+If you have `dotenv` in your Gemfile, i18n-tasks will automatically load environment variables from `.env` files
+before executing commands. This means you can store your API keys in a `.env` file:
+
+```bash
+# .env
+GOOGLE_TRANSLATE_API_KEY=your_google_api_key
+DEEPL_AUTH_KEY=your_deepl_api_key
+OPENAI_API_KEY=your_openai_api_key
+```
+
+The dotenv integration works seamlessly - no additional configuration is required. If `dotenv` is not available,
+i18n-tasks will continue to work normally using system environment variables.
+
 <a name="google-translation-config"></a>
 ### Google Translate
 
@@ -435,6 +464,7 @@ Put the key in `GOOGLE_TRANSLATE_API_KEY` environment variable or in the config
 ```yaml
 # config/i18n-tasks.yml
 translation:
+  backend: google
   google_translate_api_key: <Google Translate API key>
 ```
 
@@ -447,11 +477,12 @@ GOOGLE_TRANSLATE_API_KEY=<Google Translate API key>
 <a name="deepl-translation-config"></a>
 ### DeepL Pro Translate
 
-`i18n-tasks translate-missing` requires a DeepL Pro API key, get it at [DeepL](https://www.deepl.com/pro).
+`i18n-tasks translate-missing` requires a DeepL Pro API key, get it at [DeepL](https://www.deepl.com/pro). You can specify alias locales if you only use the simple locales internally.
 
 ```yaml
 # config/i18n-tasks.yml
 translation:
+  backend: deepl
   deepl_api_key: <DeepL Pro API key>
   deepl_host: <optional>
   deepl_version: <optional>
@@ -460,12 +491,15 @@ translation:
     - 2c6415be-1852-4f54-9e1b-d800463496b4
   deepl_options:
     formality: prefer_less
+  deepl_locale_aliases:
+    en: en-us
+    pt: pt-br
 ```
 
 or via environment variables:
 
 ```bash
-DEEPL_API_KEY=<DeepL Pro API key>
+DEEPL_AUTH_KEY=<DeepL Pro API key>
 DEEPL_HOST=<optional>
 DEEPL_VERSION=<optional>
 ```
@@ -478,6 +512,7 @@ DEEPL_VERSION=<optional>
 ```yaml
 # config/i18n-tasks.yml
 translation:
+  backend: yandex
   yandex_api_key: <Yandex API key>
 ```
 
@@ -495,6 +530,7 @@ YANDEX_API_KEY=<Yandex API key>
 ```yaml
 # config/i18n-tasks.yml
 translation:
+  backend: openai
   openai_api_key: <OpenAI API key>
   openai_model: <optional>
 ```
@@ -506,6 +542,69 @@ OPENAI_API_KEY=<OpenAI API key>
 OPENAI_MODEL=<optional>
 ```
 
+<a name="watsonx-translation-config"></a>
+### watsonx Translate
+
+`i18n-tasks translate-missing` requires a watsonx project and api key, get it at [IBM watsonx](https://www.ibm.com/watsonx/).
+
+```yaml
+# config/i18n-tasks.yml
+translation:
+  backend: watsonx
+  watsonx_api_key: <watsonx API key>
+  watsonx_project_id: <watsonx project id>
+  watsonx_model: <optional>
+```
+
+or via environment variable:
+
+```bash
+WATSONX_API_KEY=<watsonx API key>
+WATSONX_PROJECT_ID=<watsonx project id>
+WATSONX_MODEL=<optional>
+```
+
+### Prism-based scanner for Ruby and ERB files
+
+There is a scanner based on [Prism](https://github.com/ruby/prism) usable in two different modes.
+
+- `rails` mode parses Rails code and handles context such as controllers, before_actions, model translations and more.
+- `ruby` mode parses Ruby code only, and works similar to the existing whitequark/parser-implementation.
+- The parser is used for both ruby and ERB files.
+
+#### `rails` mode
+
+It handles the following cases:
+- Translations called in `before_actions`
+- Translations called in nested methods
+- `Model.human_attribute_name` calls
+- `Model.model_name.human` calls
+
+Enabled it by adding:
+```yaml
+search:
+  prism: "rails"
+```
+
+to your `config/i18n-tasks.yml` file.
+
+#### `ruby` mode
+
+It finds all `I18n.t`, `I18n.translate`, `t` and `translate` calls in Ruby code. Enabled it by adding:
+
+```yaml
+search:
+  prism: "ruby"
+```
+
+The goal is to replace the whitequark/parser-based scanner with this one in the future.
+
+#### Help us out with testing
+
+Please install the latest version of the gem and run `i18n-tasks check-prism` which will parse everything with the whitequark/parser-based scanner and then everything with the Prism-scanner and try to compare the results.
+
+Open up issues with any parser crashes, missed translations or false positives.
+
 ## Interactive console
 
 `i18n-tasks irb` starts an IRB session in i18n-tasks context. Type `guide` for more information.

data/Rakefile

@@ -1,13 +1,13 @@
 # frozen_string_literal: true
 
-require 'bundler/gem_tasks'
-require 'rspec/core/rake_task'
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
 RSpec::Core::RakeTask.new(:rspec)
 task default: :rspec
 
 task :irb do
   # $: << File.expand_path('lib', __FILE__)
-  require 'i18n/tasks'
-  require 'i18n/tasks/commands'
+  require "i18n/tasks"
+  require "i18n/tasks/commands"
   I18n::Tasks::Commands.new(I18n::Tasks::BaseTask.new).irb
 end

data/bin/i18n-tasks

@@ -1,15 +1,15 @@
 #!/usr/bin/env ruby
 # frozen_string_literal: true
 
-require_relative '../spec/bin_simplecov_helper' if ENV['I18N_TASKS_BIN_SIMPLECOV_COVERAGE']
+require_relative "../spec/bin_simplecov_helper" if ENV["I18N_TASKS_BIN_SIMPLECOV_COVERAGE"]
 
 # prevent i18n gem warning
-require 'i18n'
+require "i18n"
 i18n_gem_config = I18n.config
 if i18n_gem_config.respond_to?(:enforce_available_locales=) && i18n_gem_config.enforce_available_locales.nil?
   i18n_gem_config.enforce_available_locales = true
 end
 
-require 'i18n/tasks/cli'
+require "i18n/tasks/cli"
 
 I18n::Tasks::CLI.start(ARGV)

data/config/locales/en.yml

@@ -26,7 +26,7 @@ en:
           strict: >-
             Avoid inferring dynamic key usages such as t("cats.#{cat}.name"). Takes precedence over
             the config setting if set.
-          translation_backend: Translation backend (google or deepl)
+          translation_backend: Translation backend [google, deepl, yandex, openai])
           value: >-
             Value. Interpolates: %%{value}, %%{human_key}, %%{key}, %%{default}, %%{value_or_human_key},
             %%{value_or_default_or_human_key}
@@ -34,6 +34,9 @@ en:
         add_missing: add missing keys to locale data, optionally match a pattern
         check_consistent_interpolations: verify that all translations use correct interpolation variables
         check_normalized: verify that all translation data is normalized
+        check_prism: verify that Prism-scanner produces the same (or better) results as the default
+          parser
+        check_reserved_interpolations: verify that all translations avoid reserved interpolation variables
         config: display i18n-tasks configuration
         cp: copy the keys in locale data that match the given pattern
         data: show locale data
@@ -69,6 +72,7 @@ en:
       enum_opt:
         invalid: "%{invalid} is not one of: %{valid}."
       errors:
+        invalid_backend: 'Invalid backend: %{invalid}. Must be one of %{valid}.'
         invalid_format: 'invalid format: %{invalid}. valid: %{valid}.'
         invalid_locale: 'invalid locale: %{invalid}'
         invalid_missing_type:
@@ -123,12 +127,24 @@ en:
         other: "%{count} translation will be removed from %{locales}."
       noop: No unused keys to remove
       removed: Removed %{count} keys
+    reserved_interpolations:
+      details_title: Reserved interpolation keys
+      none: No reserved interpolations found.
     translate_missing:
       translated: Translated %{count} keys
     unused:
       none: Every translation is in use.
     usages:
       none: No key usages found.
+    watsonx_translate:
+      errors:
+        no_api_key: >-
+          Set watsonx API key via WATSONX_API_KEY environment variable or translation.watsonx_api_key
+          in config/i18n-tasks.yml. Get the key at https://www.ibm.com/products/watsonx-ai.
+        no_project_id: >-
+          Set watsonx Project ID via WATSONX_PROJECT_ID environment variable or translation.watsonx_api_key
+          in config/i18n-tasks.yml. Get the key at https://www.ibm.com/products/watsonx-ai.
+        no_results: watsonx returned no results.
     yandex_translate:
       errors:
         no_api_key: >-

data/config/locales/ru.yml

@@ -23,7 +23,7 @@ ru:
           out_format: 'Формат вывода: %{valid_text}.'
           pattern_router: 'Использовать pattern_router: ключи распределятся по файлам согласно data.write'
           strict: Не угадывать динамические использования ключей, например `t("category.#{category.key}")`
-          translation_backend: Движок перевода (google или deepl)
+          translation_backend: Движок перевода [google, deepl, yandex, openai]
           value: >-
             Значение, интерполируется с %%{value}, %%{human_key}, %%{key}, %%{default}, %%{value_or_human_key},
             %%{value_or_default_or_human_key}
@@ -32,6 +32,10 @@ ru:
         check_consistent_interpolations: убедитесь, что во всех переводах используются правильные
           интерполяционные переменные
         check_normalized: проверить, что все файлы переводов нормализованы
+        check_prism: убедитесь, что Prism-scanner выдает те же (или лучшие) результаты, что и анализатор
+          по умолчанию
+        check_reserved_interpolations: проверьте, что все переводы обходятся без зарезервированных
+          интерполяционных переменных
         config: показать конфигурацию
         cp: скопируйте ключи в данных локали, соответствующие заданному шаблону
         data: показать данные переводов
@@ -66,6 +70,7 @@ ru:
       enum_opt:
         invalid: "%{invalid} не является одним из: %{valid}."
       errors:
+        invalid_backend: 'Недопустимый источник данных: %{invalid}. Должен быть одним из %{valid}.'
         invalid_format: 'Неизвестный формат %{invalid}. Форматы: %{valid}.'
         invalid_locale: Неверный язык %{invalid}
         invalid_missing_type:
@@ -124,12 +129,24 @@ ru:
         other: Переводы (%{count}) будут удалены из %{locales}.
       noop: Нет неиспользуемых ключей
       removed: Удалены ключи (%{count})
+    reserved_interpolations:
+      details_title: Зарезервированные ключи интерполяции
+      none: NЗарезервированных интерполяций не обнаружено.
     translate_missing:
       translated: Переведены ключи (%{count})
     unused:
       none: Все переводы используются.
     usages:
       none: Не найдено использований.
+    watsonx_translate:
+      errors:
+        no_api_key: >-
+          Установите ключ API watsonx через переменную среды WATSONX_API_KEY или translation.watsonx_api_key
+          в config/i18n-tasks.yml. Получите ключ на https://www.ibm.com/products/watsonx-ai.
+        no_project_id: >-
+          Установите идентификатор проекта watsonx через переменную среды WATSONX_PROJECT_ID или translation.watsonx_api_key
+          в config/i18n-tasks.yml. Получите ключ на https://www.ibm.com/products/watsonx-ai.
+        no_results: watsonx не вернул результатов.
     yandex_translate:
       errors:
         no_api_key: |-

data/i18n-tasks.gemspec

@@ -1,17 +1,17 @@
 # frozen_string_literal: true
 
-lib = File.expand_path('lib', __dir__)
+lib = File.expand_path("lib", __dir__)
 $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
-require 'i18n/tasks/version'
+require "i18n/tasks/version"
 
 Gem::Specification.new do |s|
-  s.name          = 'i18n-tasks'
-  s.version       = I18n::Tasks::VERSION
-  s.authors       = ['glebm']
-  s.email         = ['glex.spb@gmail.com']
-  s.license       = 'MIT'
-  s.summary       = 'Manage localization and translation with the awesome power of static analysis'
-  s.description   = <<~TEXT
+  s.name = "i18n-tasks"
+  s.version = I18n::Tasks::VERSION
+  s.authors = ["glebm"]
+  s.email = ["glex.spb@gmail.com"]
+  s.license = "MIT"
+  s.summary = "Manage localization and translation with the awesome power of static analysis"
+  s.description = <<~TEXT
     i18n-tasks helps you find and manage missing and unused translations.
 
     It analyses code statically for key usages, such as `I18n.t('some.key')`, in order to report keys that are missing or unused,
@@ -25,39 +25,29 @@ Gem::Specification.new do |s|
     # Or for minitest:
     cp $(bundle exec i18n-tasks gem-path)/templates/minitest/i18n_test.rb test/
   TEXT
-  s.homepage = 'https://github.com/glebm/i18n-tasks'
+  s.homepage = "https://github.com/glebm/i18n-tasks"
   s.metadata = {
-    'issue_tracker' => 'https://github.com/glebm/i18n-tasks',
-    'rubygems_mfa_required' => 'true'
+    "changelog_uri" => "https://github.com/glebm/i18n-tasks/blob/main/CHANGES.md",
+    "issue_tracker" => "https://github.com/glebm/i18n-tasks",
+    "rubygems_mfa_required" => "true",
+    "source_code_uri" => "https://github.com/glebm/i18n-tasks"
   }
-  s.required_ruby_version = '>= 2.6', '< 4.0'
+  s.required_ruby_version = ">= 3.1"
 
   s.files = `git ls-files`.split($/)
   s.files -= s.files.grep(%r{^(doc/|\.|spec/)}) + %w[CHANGES.md config/i18n-tasks.yml Gemfile]
-  s.executables   = s.files.grep(%r{^bin/}) { |f| File.basename(f) } - %w[i18n-tasks.cmd]
-  s.require_paths = ['lib']
+  s.executables = s.files.grep(%r{^bin/}) { |f| File.basename(f) } - %w[i18n-tasks.cmd]
+  s.require_paths = ["lib"]
 
-  s.add_dependency 'activesupport', '>= 4.0.2'
-  s.add_dependency 'ast', '>= 2.1.0'
-  s.add_dependency 'erubi'
-  s.add_dependency 'highline', '>= 2.0.0'
-  s.add_dependency 'i18n'
-  s.add_dependency 'parser', '>= 3.2.2.1'
-  s.add_dependency 'rails-i18n'
-  s.add_dependency 'rainbow', '>= 2.2.2', '< 4.0'
-  s.add_dependency 'terminal-table', '>= 1.5.1'
-  s.add_development_dependency 'bundler', '~> 2.0', '>= 2.0.1'
-  s.add_development_dependency 'overcommit', '~> 0.58.0'
-  s.add_development_dependency 'rake'
-  s.add_development_dependency 'rspec', '~> 3.3'
-  s.add_development_dependency 'rubocop', '~> 1.50.1'
-  s.add_development_dependency 'rubocop-rake', '~> 0.6.0'
-  s.add_development_dependency 'rubocop-rspec', '~> 2.19.0'
-  s.add_development_dependency 'simplecov'
-  s.add_development_dependency 'yard'
-
-  # Translation backends
-  s.add_development_dependency 'deepl-rb', '>= 2.1.0'
-  s.add_development_dependency 'easy_translate', '>= 0.5.1' # Google Translate
-  s.add_development_dependency 'yandex-translator', '>= 0.3.3'
+  s.add_dependency "activesupport", ">= 4.0.2"
+  s.add_dependency "ast", ">= 2.1.0"
+  s.add_dependency "erubi"
+  s.add_dependency "highline", ">= 3.0.0"
+  s.add_dependency "i18n"
+  s.add_dependency "parser", ">= 3.2.2.1"
+  s.add_dependency "prism"
+  s.add_dependency "rails-i18n"
+  s.add_dependency "rainbow", ">= 2.2.2", "< 4.0"
+  s.add_dependency "ruby-progressbar", "~> 1.8", ">= 1.8.1"
+  s.add_dependency "terminal-table", ">= 1.5.1"
 end

data/lib/i18n/tasks/base_task.rb

@@ -1,24 +1,24 @@
 # frozen_string_literal: true
 
-require 'i18n/tasks/command_error'
-require 'i18n/tasks/split_key'
-require 'i18n/tasks/key_pattern_matching'
-require 'i18n/tasks/logging'
-require 'i18n/tasks/plural_keys'
-require 'i18n/tasks/references'
-require 'i18n/tasks/html_keys'
-require 'i18n/tasks/used_keys'
-require 'i18n/tasks/ignore_keys'
-require 'i18n/tasks/missing_keys'
-require 'i18n/tasks/interpolations'
-require 'i18n/tasks/unused_keys'
-require 'i18n/tasks/translation'
-require 'i18n/tasks/locale_pathname'
-require 'i18n/tasks/locale_list'
-require 'i18n/tasks/string_interpolation'
-require 'i18n/tasks/data'
-require 'i18n/tasks/configuration'
-require 'i18n/tasks/stats'
+require "i18n/tasks/command_error"
+require "i18n/tasks/split_key"
+require "i18n/tasks/key_pattern_matching"
+require "i18n/tasks/logging"
+require "i18n/tasks/plural_keys"
+require "i18n/tasks/references"
+require "i18n/tasks/html_keys"
+require "i18n/tasks/used_keys"
+require "i18n/tasks/ignore_keys"
+require "i18n/tasks/missing_keys"
+require "i18n/tasks/interpolations"
+require "i18n/tasks/unused_keys"
+require "i18n/tasks/translation"
+require "i18n/tasks/locale_pathname"
+require "i18n/tasks/locale_list"
+require "i18n/tasks/string_interpolation"
+require "i18n/tasks/data"
+require "i18n/tasks/configuration"
+require "i18n/tasks/stats"
 
 module I18n
   module Tasks