translation_diff 1.0.0

data/docs/how-it-works.md

@@ -0,0 +1,145 @@
+# How it works
+
+A call to `TranslationDiff.translate` walks a value, cuts the prose in it
+into sentences, translates only the sentences no cache already holds, and
+puts the value back together in the shape it arrived in.
+
+`TranslationDiff::Translator` coordinates document assembly, provider
+resolution, settling the source language, language validation and the
+sentence cache. `TranslationDiff::Dispatcher` takes over once there are cache
+misses to send: it packs them into batches, throttles each one, makes the
+wire call, and fires the `request`, `rate_limit` and `usage` events.
+Everything below is a collaborator one of the two drives.
+
+## The steps
+
+1. **The caller's structure is walked, not flattened.**
+   `TranslationDiff::Document` visits every leaf `String` of a String, Array
+   or deep Hash and can rebuild the same shape from new leaves.
+   `TranslationDiff::Leaves` holds the two promises the structure itself does
+   not: a nested `nil` comes back as `""`, and the `values` count in an event
+   payload is every leaf the caller wrote, translatable or not.
+
+2. **Each leaf becomes a passage of markup and prose.**
+   `TranslationDiff::Passage` parses the string with `ox` and records where
+   every construct begins, so each run of the source is either markup --
+   tags, comments, CDATA, doctypes, processing instructions, the bodies of
+   `config.opaque_elements` (`script`, `style`, `pre` and `code` by
+   default, matched case-insensitively), and anything inside
+   `class="notranslate"` -- or prose.
+   Each run becomes a `TranslationDiff::Fragment`, and a fragment is always a
+   slice of the source, never a rebuilt string.
+
+3. **Prose is cut into sentences.**
+   A prose fragment is split by `config.segmenter` (see [The segmenter
+   contract](contracts.md#the-segmenter-contract)) into
+   `TranslationDiff::Segment`s. A segment keeps the whitespace it was found
+   in: its `#core` is the text a provider sees, with entity references
+   decoded to the characters they mean, and its `#render` is markup again.
+
+4. **The cache is consulted once, for every sentence at once.**
+   `TranslationDiff::SentenceCache` builds one key per sentence and reads
+   them all in a single `read_multi`. Sentences it answers for are already
+   done; the rest are misses. See [Caching](caching.md).
+
+5. **The misses are handed to `TranslationDiff::Dispatcher`.**
+   `TranslationDiff::Batch.pack` groups them into batches that fit inside the
+   provider's declared `max_batch_size` and `max_request_size` -- its
+   `TranslationDiff::Capabilities` (see [Providers](providers.md)).
+   `Dispatcher` throttles each batch through `config.rate_limiter_instance`
+   when one is configured, sends it to the provider, and applies the reply
+   back onto the very segments that produced it -- no step ever correlates a
+   translation to a sentence by position after the fact.
+
+6. **What came back is written home, and the value is rebuilt.**
+   Only sentences that actually got a translation are cached, through
+   `TranslationDiff::SentenceCache#store` -- see [`write_multi` is
+   optional](caching.md#write_multi-is-optional) for what happens when the
+   store's write fails partway through a batch. Then each passage renders
+   itself -- markup fragments byte-exact, translated sentences re-encoded as
+   HTML text -- and `Document` puts the renders back into the caller's
+   shape.
+
+`TranslationDiff::Markup` is the small module underneath steps 2, 3 and 6: it
+decodes entity references on the way to a provider and, in
+`TranslationDiff::Translation::Response.build`, on the way back too, for
+every provider -- Google and DeepL both return HTML-escaped text, and
+without the second decode a vendor's own `&` was escaped a second time, so
+`didn't` came back as `didn't`. Named entities, and both the decimal
+(`'`) and hex (`'`) numeric forms, are decoded; an entity neither
+decoder recognizes, or one that would decode to invalid UTF-8, is left
+exactly as it arrived.
+
+Decoding a reply raw would make `&lt;` a bare `<`, and `ox` reads a bare `<`
+in front of a letter as an opening tag -- a provider's own `&lt;b attack`
+would become a real `<b attack>` element. So a reply is escaped the same way
+a source document's own bare angles already are, before it is decoded, and
+`Segment#render` re-encodes a translated sentence with
+`Markup.encode_translation`: `&` is always escaped, and so is a `<` that is
+not shaped like a tag -- a source document's own bare `<` is untouched by
+this. **This is a behaviour change:** `if a < b then stop.` used to come
+back with the bare `<` exactly as written; it now comes back
+`if a &lt; b then stop.`, the correct HTML encoding of that character,
+rendering identically in a browser but visible to anything comparing output
+byte-for-byte against an earlier release. `>` is left alone -- a stray `>`
+never opens anything a parser would honour, so there is nothing to protect
+it from.
+
+*NOTE:* if `:from` is not specified or equal to nil, then the provider's `#detect` will be called once with a sample of text up to 100 characters long to determine the language, and `#translate` will be called separately with the entire text.
+        Try to specify `:from` explicitly to save the extra call -- it also improves segmentation, since the segmenter only sees a language when `:from` is given (see [The segmenter contract](contracts.md#the-segmenter-contract)).
+
+## Input
+
+`TranslationDiff.translate` can receive string, array or deep hash and will return the same, but translated.
+
+```ruby
+TranslationDiff.translate("test", from: "en", to: "es")
+TranslationDiff.translate(%w[test language], from: "en", to: "es")
+TranslationDiff.translate(
+  { title: "test", values: { type: "frequent" } }, from: "en", to: "es"
+)
+```
+
+A leaf that is not a `String` -- a number, a symbol, `true` -- is handed back
+untouched. A nested `nil` comes back as `""`. See
+`TranslationDiff::Document` and `TranslationDiff::Leaves` for details.
+
+`to:` is not optional. It defaults to `nil` in the signature and a `nil`
+target raises `ArgumentError` naming the keyword, rather than silently
+handing your values back untranslated.
+
+## HTML
+
+You can pass HTML as like as plain text:
+
+```ruby
+TranslationDiff.translate("<b>Black</b>", from: "en", to: "es")
+```
+
+A `<pre>` or `<code>` block is not prose to this gem, so it is left alone.
+`config.opaque_elements` names the set treated this way -- `script`,
+`style`, `pre` and `code` by default -- matched case-insensitively, so
+`<PRE><CODE>` and `<STYLE>` (the shape Word, Outlook and older CMSes emit)
+are opaque too. An application can widen or narrow the set. Measured
+against the live Google API:
+
+```ruby
+TranslationDiff.translate(
+  "<pre><code>curl -s https://example.com/level | jq '.meters'</code></pre>",
+  from: "en", to: "es"
+)
+# => "<pre><code>curl -s https://example.com/level | jq '.meters'</code></pre>"
+```
+
+Before `pre` and `code` joined the opaque set, nothing told the pipeline
+that code holds language, not prose: the segmenter cut the block above at
+the quote and handed `meters'` to the provider as a sentence of its own,
+and Google translated it -- `jq '.meters'` came back as `jq '.metros'`,
+inside the quoted filter. Wrap a block you don't want touched in
+`class="notranslate"` instead, for protection finer than an element, or for
+an element outside `config.opaque_elements`; the providers that honour it
+(see [Providers](providers.md)) leave it exactly as written.
+
+**Upgrading:** widening what counts as markup changes what gets sent to the
+provider, so it changes cache keys for any document containing a `pre` or
+`code` element -- see [Caching](caching.md#what-a-cache-key-is-made-of).

data/docs/instrumentation.md

@@ -0,0 +1,96 @@
+# Instrumentation and logging
+
+`config.instrumenter` accepts anything satisfying
+`ActiveSupport::Notifications`' interface -- `#instrument(name, payload) { }` --
+and `config.logger` accepts a standard `Logger`. Neither is required: with
+both unset, `TranslationDiff.translate` runs exactly the same, at no extra
+cost.
+
+A translation emits up to six events, each named `<name>.translation_diff`:
+
+| Event | Fired | Payload |
+| --- | --- | --- |
+| `translate` | Once per `translate` call that has something to translate, wrapping the whole thing -- including a call served entirely from cache, which never reaches the provider. A call whose source and target languages are the same, or whose values hold no translatable text at all, returns early and emits no events. | `call_id`, `from`, `to`, `provider`, `values` (number of texts), `characters` (total considered by this call) |
+| `cache` | Once per such call, after checking the cache for every sentence at once -- whether or not anything is left to send the provider. | `call_id`, `provider`, `hits`, `misses` |
+| `request` | Once per batch actually sent to the provider (skipped entirely on a full cache hit). | `call_id`, `provider`, `batch` (values sent), `characters` (sent by this batch) |
+| `rate_limit` | Once per batch sent to the provider, only when a rate limiter is configured. | `call_id`, `provider`, `characters` |
+| `usage` | Once per batch actually sent to the provider, right after `request`. | `call_id`, `provider`, `characters`, `billed_characters`, `reported`, `model` |
+| `cache_error` | Only when writing the translation back to the cache fails -- after the provider has already answered. Never fires on a successful write, so it is not part of every call the way the other five are. | `call_id`, `provider`, `error` (the failed write's error class, as a string) |
+
+**Every event above carries `call_id`.** It is generated once per
+`translate` call, opaque, and never derived from the text. Before it, a
+subscriber receiving `cache`, `request`, `rate_limit`, `usage` or
+`cache_error` events had no way to tell which `translate` call any of them
+belonged to, short of tagging `Thread.current` itself -- a workaround that
+breaks the moment two translations share a thread.
+
+**`translate`'s `characters` and `request`'s `characters` measure different
+things.** `translate`'s is the total this call considered -- every
+non-blank sentence, hit or miss, whether or not any of it was sent to the
+provider -- so a call served entirely from cache still reports a number
+instead of nothing, even though no `request` event fires for it at all.
+`request`'s keeps its narrower meaning: what this one batch actually sent.
+`usage`'s `characters` carries that same narrower meaning too, per batch,
+like `request`'s. Same name, three events, two meanings -- a subscriber
+summing the wrong one gets a wrong bill.
+
+`cache_error` is what a failing cache write looks like from the outside:
+the write itself is rescued, not the translation, which still reaches the
+caller -- see
+[The three write paths fail differently](caching.md#the-three-write-paths-fail-differently).
+`error` is the exception's class name, never its message, which could echo
+the row it failed to write. Which class you see depends on the store: a
+store that redacts its own failures reports that redaction, so
+`Stores::ActiveRecord` always gives `"TranslationDiff::Error"` -- the
+adapter's own class is named inside that error's (content-free) message,
+not in this payload. `Stores::Redis` does not wrap, so it gives the
+driver's class, `"Redis::CannotConnectError"` and the like. Alert on the
+event, not on a particular class name.
+
+The same failure is logged at **warn**, not debug: an application whose
+cache has quietly stopped accepting writes pays the provider for every
+sentence, every time, and a signal only visible at debug level is one
+nobody sees in production.
+
+`usage`'s `characters` is what this library sent, counted locally -- the same
+number `request` carries. `billed_characters` is what the provider said it
+charged for, or `nil` when it said nothing. **`reported` means the provider
+reports billing at all -- not that this particular response was billed.**
+`billed_characters: nil` alone cannot tell "this provider never says" apart
+from "this response omitted it"; `reported` is what makes the `nil` honest.
+Summing `billed_characters` across providers without checking `reported`
+first produces a total that is quietly too low, since only three of the six
+built-in providers (DeepL, Azure, ModernMT) report billing at all -- the
+other three always answer `nil`. `model` is the model the provider used,
+when it names one, and `nil` otherwise.
+
+**`cache` fires once per call as of translation_diff 1.0.0, not once per chunk.** The cache is
+now consulted for every sentence in one `read_multi` before anything is
+batched, so there is one event where there used to be one per chunk. `hits`
+and `misses` still sum to the same totals over a call, so a counter that adds
+them up is unaffected; a counter of *events*, or a histogram of per-chunk hit
+ratios, will see the cardinality drop.
+
+**Instrumentation payloads never contain the text being translated, its
+translation, or a credential.** This is a guarantee, not an implementation
+detail: this library handles other people's content, and an instrumenter
+usually writes somewhere that content must not go. Only counts, language
+codes and provider names cross that boundary.
+
+`config.logger` receives one `debug` line per provider resolution, naming
+the provider class in use -- nothing about the content being translated. The
+same guarantee applies to it as to instrumentation payloads: no log line this
+library writes carries the text being translated, its translation, or a
+credential.
+
+**No HTTP-backed provider ever receives `config.logger`, and there is no way
+to opt one in.** `TranslationDiff::HTTPProvider` installs no logging
+middleware on its Faraday connection and never passes a logger to it -- this
+is enforced by `test/support/http_provider_contract.rb`, not merely
+documented. Earlier versions wrapped `deepl-rb`, which logged a
+`Request details:` line at DEBUG holding the full
+`Authorization: DeepL-Auth-Key ...` header and the request payload -- your
+API key and the text being translated -- if you gave it a logger of its own.
+Owning the transport directly closed that door rather than working around
+it: nothing this library builds writes source text, a translation, or a
+credential anywhere, and no configuration option reopens that.

data/docs/languages.md

@@ -0,0 +1,94 @@
+# Languages
+
+`TranslationDiff::Languages` answers, without making a request, whether a
+provider translates a given source into a given target. `Translator` calls it
+before every translation and refuses a pair it says no to.
+
+```ruby
+TranslationDiff::Languages.supports?(:deepl, from: "en", to: "ru")   # => true
+TranslationDiff::Languages.supports?(:amazon, from: "en", to: "ru")  # => nil
+```
+
+Three answers, not two: `true`, `false`, and `nil` for "no opinion". `nil` is
+not a refusal -- a provider this registry knows nothing about is never
+blocked from translating anything.
+
+## What ships
+
+One JSON file per provider under `data/languages/`, each naming the date it
+was captured and the endpoint it came from:
+
+```json
+{
+  "provider": "deepl",
+  "captured_at": "2026-09-10",
+  "endpoint": "https://api.deepl.com/v2/languages",
+  "source": ["ar", "bg", "cs", "..."],
+  "target": ["ar", "bg", "cs", "en-gb", "en-us", "..."]
+}
+```
+
+`captured_at` is not decoration: it is the difference between "these are the
+languages" and "these were the languages on 10 September 2026", and only the
+second is true.
+
+Data ships for **DeepL, Google, Azure and ModernMT** only. **Amazon** ships
+none -- its language list needs signed credentials, so nothing can be fetched
+without them. **LibreTranslate** ships none either, for a different reason:
+it is self-hosted, so the language set belongs to whichever instance you
+point this gem at, not to a vendor this gem can capture once and ship.
+
+A provider with no shipped data -- Amazon, LibreTranslate, or any provider of
+your own -- refuses nothing. `Languages.supports?` returns `nil` for it,
+every time, and `Translator` treats `nil` the same as `true`.
+
+## Matching a code
+
+A code is matched downcased, on its primary subtag, so `en-GB` and `en`
+match each other in both directions.
+
+This is deliberately permissive. The registry exists to catch a wrong
+language -- `to: "klingon"`, a swapped pair, a typo -- not to adjudicate a
+wrong regional variant. `Languages.supports?(:azure, from: "en", to: "zh")`
+answers `true`, even though Azure itself wants `zh-Hans` or `zh-Hant` as a
+target and would reject a bare `zh`. The rule can over-allow; it must never
+wrongly refuse a pair the provider would actually have accepted.
+
+One more rule, for a mismatch the vendors themselves create. ModernMT
+publishes ISO 639-3 individual codes where applications write the ISO 639-1
+macrolanguage: it lists `pes`, not `fa`, and `uzn`, not `uz`. A vendor that
+translates the individual language translates the macrolanguage it belongs
+to, so `to: "fa"` is accepted against a list carrying `pes`.
+
+That holds in one direction only. A vendor publishing `zh` has not promised
+to accept `cmn`, so `Languages.supports?(:deepl, from: "en", to: "cmn")`
+answers `false` -- DeepL would reject the code it was sent.
+
+## The two escapes
+
+Shipped data goes stale, and a vendor adding a language must not make this
+gem refuse work that would now succeed. Two escapes exist for that:
+
+```ruby
+# Per call
+TranslationDiff.translate(text, from: "en", to: "yue", assume_supported: true)
+
+# Globally
+TranslationDiff.configure { |config| config.validate_languages = false }
+```
+
+`assume_supported:` is a reserved keyword, like `provider:` and `config:` --
+it is read by `Translator` and never forwarded to the provider itself.
+
+## `rake languages:refresh`
+
+A maintainer's tool, not something an application runs: it re-fetches every
+provider's language lists from its vendor and rewrites the shipped files,
+which is why it needs each vendor's own credentials configured to do
+anything.
+
+A provider whose fetch fails keeps its previous file -- an emptied list from
+a timed-out request would be worse than a stale one -- and the task reports
+which providers failed and why, rather than failing silently. Its output is
+meant to be reviewed as a diff, the way any change to shipped data should be,
+before it is committed.