i18n-keyless-rails 3.7.0 → 3.8.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 41fc3ec59089f3a6e65ecc8b3d32483a32a69821580090b1c71c1ad7fbe2b078
-  data.tar.gz: ceb6a6ee2b421d697a13c2a8b5333f3f787eccd9e8589bb596bbcd8765797cb5
+  metadata.gz: 0b5f7bbea883a39be3f717ee063ceb055b4876f7ce0a8027e15d0df728b9d1fe
+  data.tar.gz: afe3e6d4863e6a4867e9a412171bc8e4893def9b9fc0f0aa36ffda03cda13186
 SHA512:
-  metadata.gz: e55140376f1b50deab0bf7f22df1f0a57fbda948113520cf6eb54adc4a0957d282dab930428e3c711a8798ef26662acd5fee19746c5a88fe603bd410b16e8219
-  data.tar.gz: 48a0040816d435284bf1eb6dce88a2c4a77a3e51657d9d51155b74bba223ea1c657cbc482cfa20205d16d43808b855f279d7cc6f06903ece92781de09687b719
+  metadata.gz: 1ddf136839855b5e00bcbd685bebb6ce77bfe790454b20dd3f05255114506d10d4adef4e6adecf24dad17feab9fd4e24fc70bae3623d017822e51b226ab4987a
+  data.tar.gz: 7222d1ba284586cf64d14ff644e9507050d40c81cc497f00cc104222df6d5226567e92b07011ab89d3710174afc9f9070b6e6430b39a68a3cb142544395054fd

data/README.md

@@ -104,6 +104,7 @@ end
 | `queue` | `I18N_KEYLESS_QUEUE` | none | An ActiveJob queue name: the misses of a request are enqueued as one `I18nKeyless::TranslateMissingKeysJob` instead of being sent after the response. |
 | `logger` | | `Rails.logger` | Where the `i18n-keyless:` warnings go. |
 | `rails_key_pattern` | | see above | The Rails-key rule. `nil`: every string is keyless. |
+| `bundle_path` | `I18N_KEYLESS_BUNDLE_PATH` | none | The directory of a precompiled bundle (`manifest.json` plus `<namespace>/<lang>.json`). Every dictionary it covers is read at boot, never fetched. See "Precompiled bundle". |
 
 `I18N_KEYLESS_LANGUAGES` is the list the API translates a new string into, and the list it
 stores as your project's languages (it replaces the previous one, like the `supported` list
@@ -158,6 +159,18 @@ most `cache_ttl` seconds later. The miss guard and the usage lock live in the ca
 with a `:memory_store` they are per process, so use a shared store (Redis, Memcached, the
 database, the file store) in production, as Rails recommends anyway.
 
+## Precompiled bundle
+
+Export your dictionaries once, at build time, with the MCP `export_bundle` tool or
+`GET /translate/bundle`: a directory holding `manifest.json` and one
+`<namespace>/<lang>.json` per dictionary. Point `bundle_path`
+(`I18N_KEYLESS_BUNDLE_PATH`) at it, for example `Rails.root.join("i18n-keyless")`. Every
+`(namespace, lang)` the manifest covers is then read at boot, into the process and into the
+cache with the manifest's cursor, and never fetched from the API; the cache wins only when
+it already holds a newer copy of the same language (a review that landed after the export).
+Nothing else changes: a pair the manifest does not list is fetched as before, a miss still
+POSTs, and its answer is merged next to the bundled lines.
+
 ## Limitations
 
 - **Plurals.** `t('Il y a %{count} pommes', count: n)` sends one source string and I18n

data/SKILL.md

@@ -76,10 +76,17 @@ signatures; the `i18nk` helper is mixed into views, controllers, mailers and job
 `I18nKeyless.configure { |c| ... }` in an initializer, or the environment. Keys: `enabled`,
 `api_key`, `api_url`, `primary`, `languages`, `namespace`, `cache`, `cache_ttl`,
 `cache_prefix`, `timeout` (10 s), `retry` (`[500, 1500]` ms), `concurrency` (30), `usage`
-(true), `queue`, `logger`, `rails_key_pattern`.
+(true), `queue`, `logger`, `rails_key_pattern`, `bundle_path`.
 Env: `I18N_KEYLESS_ENABLED`, `I18N_KEYLESS_API_KEY`, `I18N_KEYLESS_API_URL`,
 `I18N_KEYLESS_PRIMARY_LANG`, `I18N_KEYLESS_LANGUAGES`, `I18N_KEYLESS_NAMESPACE`,
-`I18N_KEYLESS_CACHE_TTL`, `I18N_KEYLESS_USAGE`, `I18N_KEYLESS_QUEUE`.
+`I18N_KEYLESS_CACHE_TTL`, `I18N_KEYLESS_USAGE`, `I18N_KEYLESS_QUEUE`,
+`I18N_KEYLESS_BUNDLE_PATH`.
+
+Precompiled bundle: export the dictionaries at build time with the MCP `export_bundle`
+tool or `GET /translate/bundle` (a directory: `manifest.json` plus one
+`<namespace>/<lang>.json` per dictionary) and set `bundle_path` to that directory. Every
+pair the manifest covers is read at boot, never fetched; the cache wins only when it is
+newer. A miss still POSTs. Nothing else changes.
 
 ## Debug
 

data/lib/i18n_keyless/api_client.rb

@@ -36,7 +36,9 @@ module I18nKeyless
       Timeout::Error, SocketError, SystemCallError, EOFError, IOError, OpenSSL::SSL::SSLError
     ].freeze
 
-    Dictionary = Struct.new(:ok, :not_modified, :translations, :etag, :error, keyword_init: true)
+    # `last_refresh`: the API's cursor for the dictionary (`data.lastRefresh`,
+    # epoch ms as a string), kept in the cache for the bundle precedence rule.
+    Dictionary = Struct.new(:ok, :not_modified, :translations, :etag, :error, :last_refresh, keyword_init: true)
     UsageResult = Struct.new(:ok, :sent, :error, keyword_init: true)
     Outcome = Struct.new(:action, :error, :response, :json, keyword_init: true)
 
@@ -310,6 +312,8 @@ module I18nKeyless
       result.translations = translations.is_a?(Hash) ? translations.select { |_, v| v.is_a?(String) } : {}
       etag = response["ETag"].to_s
       result.etag = etag.empty? ? nil : etag
+      cursor = json.dig("data", "lastRefresh")
+      result.last_refresh = cursor.nil? || cursor.to_s.empty? ? nil : cursor.to_s
       result
     end
 

data/lib/i18n_keyless/bundle.rb

@@ -0,0 +1,137 @@
+# frozen_string_literal: true
+
+require "json"
+
+module I18nKeyless
+  # The precompiled bundle (docs/PROTOCOL.md, sections 4.5 and 7.4): a
+  # directory holding `manifest.json` and one `<namespace>/<lang>.json` per
+  # dictionary, as the MCP `export_bundle` tool or `GET /translate/bundle`
+  # yields it.
+  #
+  # A (namespace, lang) pair the manifest covers is read from the file instead
+  # of fetched, with the manifest's cursor. The two class methods `covers?`
+  # and `merge_with_storage` are pure and replayed by
+  # conformance/vectors/bundle-seed.json.
+  #
+  # Seed: { translations: Hash, last_refresh: String|nil }
+  # Stored: { translations: Hash, last_refresh: Object, lang: String }
+  class Bundle
+    attr_reader :path, :manifest
+
+    def initialize(path, manifest)
+      @path = path
+      @manifest = manifest
+    end
+
+    # Reads `manifest.json` from the directory. Nil when no path is configured;
+    # nil with one warning when the path holds no readable manifest.
+    def self.load(path, logger: nil)
+      path = path.to_s.strip.sub(%r{[/\\]+\z}, "")
+      return nil if path.empty?
+
+      manifest = read_json(File.join(path, "manifest.json"))
+      unless manifest.is_a?(Hash) && manifest["namespaces"].is_a?(Hash)
+        warn_with(logger, "bundle_path \"#{path}\" holds no readable manifest.json: the bundle is ignored")
+        return nil
+      end
+      new(path, manifest)
+    end
+
+    # `bundleCovers(manifest, namespace, lang)` of the SDKs: true when the
+    # manifest lists `lang` under `namespace`.
+    def self.covers?(manifest, namespace, lang)
+      return false unless manifest.is_a?(Hash)
+
+      languages = manifest.dig("namespaces", namespace.to_s, "languages")
+      languages.is_a?(Array) && languages.include?(lang.to_s)
+    end
+
+    # `mergeBundleWithStorage(bundle, stored, lang)` of the SDKs: the bundle is
+    # the base. What storage holds is merged on top (its keys win) and its
+    # cursor kept ONLY when its cursor is a strictly larger number than the
+    # bundle's AND it is in `lang`. A stored cursor that is empty, nil or not a
+    # number is never newer.
+    def self.merge_with_storage(bundle, stored, lang)
+      return bundle if stored.nil? || stored[:lang] != lang
+
+      stored_cursor = cursor_number(stored[:last_refresh])
+      return bundle if stored_cursor.nil?
+
+      # `Number(x)` of the SDKs: nil and "" are 0, anything else non-numeric is NaN,
+      # and nothing is larger than NaN.
+      bundle_raw = bundle[:last_refresh]
+      bundle_cursor = bundle_raw.nil? || bundle_raw.to_s.empty? ? 0.0 : cursor_number(bundle_raw)
+      return bundle if bundle_cursor.nil? || stored_cursor <= bundle_cursor
+
+      {
+        translations: bundle[:translations].merge(stored[:translations] || {}),
+        last_refresh: stored[:last_refresh]
+      }
+    end
+
+    def self.cursor_number(value)
+      return nil if value.nil?
+      return value.to_f if value.is_a?(Numeric)
+
+      text = value.to_s.strip
+      return nil if text.empty?
+
+      Float(text, exception: false)
+    end
+
+    def self.read_json(file)
+      return nil unless File.file?(file) && File.readable?(file)
+
+      JSON.parse(File.read(file))
+    rescue JSON::ParserError, SystemCallError, IOError, EncodingError
+      nil
+    end
+
+    def self.warn_with(logger, message)
+      logger&.warn("i18n-keyless: #{message}")
+    rescue StandardError
+      # Logging must never take a translation down.
+    end
+
+    def covers?(namespace, lang)
+      self.class.covers?(manifest, namespace, lang)
+    end
+
+    # The namespaces the manifest lists, in manifest order.
+    def namespaces
+      manifest["namespaces"].keys.map(&:to_s)
+    end
+
+    # The languages the manifest lists under one namespace.
+    def languages(namespace)
+      languages = manifest.dig("namespaces", namespace.to_s, "languages")
+      languages.is_a?(Array) ? languages.map(&:to_s) : []
+    end
+
+    # Every (namespace, lang) pair the manifest covers, in manifest order.
+    def pairs
+      namespaces.flat_map { |namespace| languages(namespace).map { |lang| [namespace, lang] } }
+    end
+
+    # The cursor of one namespace (epoch ms as a string), or nil.
+    def last_refresh(namespace)
+      cursor = manifest.dig("namespaces", namespace.to_s, "lastRefresh")
+      cursor.nil? ? nil : cursor.to_s
+    end
+
+    # One covered dictionary with its cursor, or nil when the pair is not
+    # covered or the file is unreadable (logged: the caller falls back to the
+    # fetch, like a pair the manifest does not list).
+    def seed(namespace, lang, logger: nil)
+      return nil unless covers?(namespace, lang)
+
+      file = File.join(path, namespace.to_s, "#{lang}.json")
+      translations = self.class.read_json(file)
+      unless translations.is_a?(Hash)
+        self.class.warn_with(logger, "bundle file \"#{file}\" is missing or unreadable: #{namespace}/#{lang} is fetched instead")
+        return nil
+      end
+      { translations: translations.select { |_, v| v.is_a?(String) }, last_refresh: last_refresh(namespace) }
+    end
+  end
+end

data/lib/i18n_keyless/config.rb

@@ -43,6 +43,12 @@ module I18nKeyless
     attr_accessor :logger
     # The Rails-key rule (see DEFAULT_RAILS_KEY_PATTERN). `nil`: every string is keyless.
     attr_accessor :rails_key_pattern
+    # The precompiled bundle: a directory holding manifest.json and one
+    # <namespace>/<lang>.json per dictionary, as the MCP `export_bundle` tool or
+    # GET /translate/bundle yields it (for example Rails.root.join("i18n-keyless")).
+    # A dictionary the manifest covers is read at boot, never fetched, unless the
+    # cache already holds a newer copy. A miss still POSTs.
+    attr_accessor :bundle_path
 
     def initialize(env = ENV)
       @enabled = truthy?(env.fetch("I18N_KEYLESS_ENABLED", "true"))
@@ -61,6 +67,7 @@ module I18nKeyless
       @queue = env["I18N_KEYLESS_QUEUE"]
       @logger = nil
       @rails_key_pattern = DEFAULT_RAILS_KEY_PATTERN
+      @bundle_path = env["I18N_KEYLESS_BUNDLE_PATH"]
     end
 
     def enabled?

data/lib/i18n_keyless/dictionary_store.rb

@@ -11,7 +11,12 @@ module I18nKeyless
   # it is served without asking the API. A stale entry is still served, and
   # revalidated with its ETag after the response (a 304 keeps it as is).
   #
-  # Entry: { translations: Hash, etag: String|nil, fetched_at: Integer, failed: Boolean }
+  # Entry: { translations: Hash, etag: String|nil, fetched_at: Integer, failed: Boolean, last_refresh: String|nil }
+  #
+  # `last_refresh` is the API's cursor for the dictionary (epoch ms as a
+  # string, PROTOCOL.md section 4.2), or the bundle's when the entry was seeded
+  # from one. It is never sent on the wire (the ETag carries freshness here):
+  # it decides the precedence between a bundle and the cache (`seed`).
   class DictionaryStore
     # Seconds a failed fetch is remembered before the API is asked again.
     FAILURE_TTL = 60
@@ -40,8 +45,41 @@ module I18nKeyless
       entry[:translations].is_a?(Hash) ? entry : nil
     end
 
-    def put(lang, namespace, translations, etag, failed: false)
-      entry = { translations: translations, etag: etag, fetched_at: Time.now.to_i, failed: failed }
+    def put(lang, namespace, translations, etag, failed: false, last_refresh: nil)
+      entry = { translations: translations, etag: etag, fetched_at: Time.now.to_i, failed: failed, last_refresh: last_refresh }
+      cache.write(key(lang, namespace), entry)
+      entry
+    end
+
+    # Applies one bundle dictionary (PROTOCOL.md 7.4) to the stored entry with
+    # the SDKs' precedence (`Bundle.merge_with_storage`): the bundle is the
+    # base and the stored slice is merged on top, its cursor kept, only when
+    # the cache is strictly newer than the bundle. Otherwise the bundle's lines
+    # win, its cursor is stored, and the entry is fresh: no fetch and no
+    # revalidation until `ttl` seconds pass. Keys the cache has and the bundle
+    # lacks (a merged POST answer) are kept either way. Idempotent: a second
+    # call with the same bundle writes nothing.
+    #
+    # @param seed [Hash] { translations: Hash, last_refresh: String|nil }
+    def seed(lang, namespace, seed)
+      stored = get(lang, namespace)
+      merged = Bundle.merge_with_storage(
+        seed,
+        stored && { translations: stored[:translations], last_refresh: stored[:last_refresh], lang: lang },
+        lang
+      )
+      translations = (stored ? stored[:translations] : {}).merge(merged[:translations])
+      storage_wins = stored && merged[:last_refresh] != seed[:last_refresh]
+      if storage_wins
+        return stored if translations == stored[:translations]
+
+        entry = stored.merge(translations: translations)
+      else
+        return stored if stored && translations == stored[:translations] && stored[:last_refresh] == merged[:last_refresh]
+
+        # The ETag named what the API answered before the export: it no longer matches this content.
+        entry = { translations: translations, etag: nil, fetched_at: Time.now.to_i, failed: false, last_refresh: merged[:last_refresh] }
+      end
       cache.write(key(lang, namespace), entry)
       entry
     end

data/lib/i18n_keyless/translator.rb

@@ -49,12 +49,15 @@ module I18nKeyless
         default_namespace: config.resolved_namespace,
         queue: config.queue,
         usage_enabled: config.usage?,
-        logger: config.resolved_logger
+        logger: config.resolved_logger,
+        bundle: Bundle.load(config.bundle_path, logger: config.resolved_logger)
       )
     end
 
+    # @param bundle [Bundle, nil] the precompiled bundle (`bundle_path`): every
+    #   (namespace, lang) it covers is seeded now, at boot, and never fetched
     def initialize(store:, api:, primary:, languages: [], default_namespace: DEFAULT_NAMESPACE, queue: nil,
-                   usage_enabled: true, logger: nil)
+                   usage_enabled: true, logger: nil, bundle: nil)
       @store = store
       @api = api
       @primary = primary
@@ -63,14 +66,18 @@ module I18nKeyless
       @queue = queue.to_s.empty? ? nil : queue.to_s
       @usage_enabled = usage_enabled
       @logger = logger
+      @bundle = bundle
       @mutex = Mutex.new
       @loaded = {}      # "lang|namespace" => lines loaded in this process
       @misses = {}      # Miss#id => Miss
       @revalidate = {}  # "lang|namespace" => [lang, namespace]
       @usage = {}       # namespace => lookup key => YYYY-MM-DD
       @warned_no_languages = false
+      seed_bundle!
     end
 
+    attr_reader :bundle
+
     def usage_enabled?
       @usage_enabled
     end
@@ -121,18 +128,20 @@ module I18nKeyless
       key
     end
 
-    # Loads the (lang, namespace) dictionary once per process.
+    # Loads the (lang, namespace) dictionary once per process. A pair the
+    # bundle covers was seeded at boot (`seed_bundle!`); after `I18n.reload!`
+    # it is seeded again here, from the file, never fetched.
     def ensure_loaded(lang, namespace)
       id = "#{lang}|#{namespace}"
       @mutex.synchronize do
         return @loaded[id] if @loaded.key?(id)
 
-        entry = store.get(lang, namespace)
+        entry = seed_from_bundle(lang, namespace) || store.get(lang, namespace)
         if entry.nil?
           # First time ever for this language: the one blocking fetch.
           result = api.fetch_dictionary(lang, namespace, nil)
           entry = if result.ok
-                    store.put(lang, namespace, result.translations, result.etag)
+                    store.put(lang, namespace, result.translations, result.etag, last_refresh: result.last_refresh)
                   else
                     store.put(lang, namespace, {}, nil, failed: true)
                   end
@@ -271,17 +280,53 @@ module I18nKeyless
       result = api.fetch_dictionary(lang, namespace, entry && entry[:etag])
       unless result.ok
         # Remember the failure briefly, so the next requests do not all retry.
-        store.put(lang, namespace, entry ? entry[:translations] : {}, entry && entry[:etag], failed: true)
+        store.put(lang, namespace, entry ? entry[:translations] : {}, entry && entry[:etag], failed: true,
+                                   last_refresh: entry && entry[:last_refresh])
         return
       end
       if result.not_modified
         store.touch(lang, namespace)
         return
       end
-      store.put(lang, namespace, result.translations, result.etag)
+      store.put(lang, namespace, result.translations, result.etag, last_refresh: result.last_refresh)
       refresh_loaded(lang, namespace, result.translations)
     end
 
+    # Boot (PROTOCOL.md 7.4, the node SDK model): every (namespace, lang) the
+    # manifest covers is read from its file into this process and into the
+    # cache, with the manifest's cursor, so the boot fetch of a covered pair
+    # never happens. The primary language is skipped: this gem never looks a
+    # primary dictionary up. An unreadable file is logged and left to the lazy
+    # path (fetched on first use), never fetched eagerly. Makes no request.
+    def seed_bundle!
+      return if bundle.nil?
+
+      @mutex.synchronize do
+        bundle.pairs.each do |namespace, lang|
+          next if lang == primary || !Locale.lang?(lang)
+
+          entry = seed_from_bundle(lang, namespace)
+          next if entry.nil?
+
+          id = "#{lang}|#{namespace}"
+          @revalidate[id] = [lang, namespace] if store.stale?(entry)
+          @loaded[id] = entry[:translations]
+        end
+      end
+    end
+
+    # The stored entry after the bundle's dictionary is applied to it (the
+    # cache wins only when it is newer), or nil when the bundle does not cover
+    # the pair or its file is unreadable. Called with the mutex held.
+    def seed_from_bundle(lang, namespace)
+      return nil if bundle.nil?
+
+      seed = bundle.seed(namespace, lang, logger: logger)
+      return nil if seed.nil?
+
+      store.seed(lang, namespace, seed)
+    end
+
     # Keeps this process's loaded lines current (a long-lived process keeps
     # the dictionaries between requests).
     def refresh_loaded(lang, namespace, lines)

data/lib/i18n_keyless/version.rb

@@ -4,5 +4,5 @@ module I18nKeyless
   # Sent as the `Version` header. The API reads its major to pick the wire
   # dialect: >= 3 means the v3 language codes ("zh-Hans", "cs"). One version
   # for every package and port of the monorepo (scripts/set-version.mjs).
-  VERSION = "3.7.0"
+  VERSION = "3.8.0"
 end

data/lib/i18n_keyless.rb

@@ -11,6 +11,7 @@ require_relative "i18n_keyless/config"
 require_relative "i18n_keyless/locale"
 require_relative "i18n_keyless/miss"
 require_relative "i18n_keyless/api_client"
+require_relative "i18n_keyless/bundle"
 require_relative "i18n_keyless/dictionary_store"
 require_relative "i18n_keyless/translator"
 require_relative "i18n_keyless/backend"

data/llms.txt

@@ -67,6 +67,7 @@ Rails-key rule (`I18nKeyless.keyless_key?`, `config.rails_key_pattern` = `/\A[a-
 | queue | I18N_KEYLESS_QUEUE | nil |
 | logger | | Rails.logger |
 | rails_key_pattern | | `/\A[a-z0-9_]+(\.[a-z0-9_]+)*\z/` (nil: every string is keyless) |
+| bundle_path | I18N_KEYLESS_BUNDLE_PATH | nil: the directory of a precompiled bundle (manifest.json + <namespace>/<lang>.json, from the MCP `export_bundle` tool or GET /translate/bundle); every covered dictionary is read at boot, never fetched; a miss still POSTs |
 
 ## Locale mapping
 

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: i18n-keyless-rails
 version: !ruby/object:Gem::Version
-  version: 3.7.0
+  version: 3.8.0
 platform: ruby
 authors:
 - Arnaud Ambroselli
@@ -54,6 +54,7 @@ files:
 - lib/i18n_keyless.rb
 - lib/i18n_keyless/api_client.rb
 - lib/i18n_keyless/backend.rb
+- lib/i18n_keyless/bundle.rb
 - lib/i18n_keyless/config.rb
 - lib/i18n_keyless/dictionary_store.rb
 - lib/i18n_keyless/helper.rb