translation_diff 1.0.0

data/docs/configuration.md

@@ -0,0 +1,205 @@
+# Configuration
+
+Everything here is configured through the one `Configuration` object yielded by `TranslationDiff.configure`.
+
+## Dependencies
+
+This gem loads four at require time: [`ox`](https://github.com/ohler55/ox) to
+walk the HTML; [`pragmatic_segmenter`](https://github.com/diasks2/pragmatic_segmenter),
+which backs the default sentence segmenter and has zero dependencies of its
+own (see [The segmenter contract](contracts.md#the-segmenter-contract)
+below if you want to avoid it); and [`faraday`](https://github.com/lostisland/faraday)
+with [`faraday-retry`](https://github.com/lostisland/faraday-retry), the HTTP
+transport every REST-backed provider (DeepL, Google, Azure, ModernMT,
+LibreTranslate) inherits and owns directly -- none of them wraps a
+vendor-supplied SDK any more.
+
+Everything else is duck typed and supplied by you: `aws-sigv4`, required
+lazily the first time the Amazon provider signs a request, with a clear
+error if it is missing. The same is true of `redis` and `connection_pool`
+once you configure `redis_url`, and of `ratelimit` on the first check once
+you configure `rate_limit`. `redis-namespace` (for the Redis cache store)
+goes one step further: this gem never requires it at all, so your
+application must `require` it itself before using the Redis-backed store.
+See [Getting started](#getting-started) below.
+
+## Getting started
+
+`deepl_api_key` is required -- the provider checks for it at build time and
+raises `TranslationDiff::ConfigurationError` naming what is missing, rather
+than failing on the first real request. Leave it unset and `DEEPL_AUTH_KEY`
+is read instead, on use rather than at load, so the variable may be exported
+after this gem is required. `redis_url` is optional: without it
+the cache lives in the process, which means the library runs before any
+infrastructure does.
+
+Every extension point takes **either a symbol naming a built-in, or an object
+of your own**:
+
+```ruby
+TranslationDiff.configure do |config|
+  config.provider = :deepl        # or any TranslationDiff::Provider of your own -- see Providers
+  config.cache = :redis           # or any object satisfying the cache store contract
+  config.segmenter = :pragmatic   # or any object satisfying the segmenter contract
+end
+```
+
+## Configuration options
+
+Every setting lives on the one `Configuration` object yielded by
+`TranslationDiff.configure`. An option not assigned falls back to its
+default; assigning it a blank string behaves as though it was never touched
+at all, so an unset environment variable never has to be special-cased.
+
+| Option | Default | Meaning |
+| --- | --- | --- |
+| `provider` | `:deepl` | The translation provider: a registered name or a `TranslationDiff::Provider` of your own. See [Providers](providers.md). |
+| `cache` | `nil` | The cache store: a registered name or an object satisfying the [cache store contract](caching.md#the-cache-store-contract). `nil` means "choose for me" -- see below. |
+| `cache_ttl` | `604_800` (one week) | Seconds an entry is kept before it expires. Read by `Stores::Redis` (a `SETEX`) and by `Stores::ActiveRecord` (written into each row's `expires_at`); `Stores::Memory` evicts by size instead and ignores it. A non-positive value (`0` or less, or `nil`) means never expires. A String is coerced, so an environment variable works; a value that is not a number is refused at `configure` time rather than mid-translation. See [SQL cache](sql-cache.md#cache_ttl-becomes-expires_at). |
+| `cache_namespace` | `"translation-diff"` | Prefix applied to every Redis key this gem writes -- both cache entries and the rate limiter's own bookkeeping. Also the `namespace` column both SQL tables share and the unit `Stores::ActiveRecord#prune` operates on. At most 64 characters -- longer is refused at `configure` time. See [SQL cache](sql-cache.md#the-tables). |
+| `cache_max_size` | `1_000` | Maximum number of entries `Stores::Memory` keeps before evicting the least recently used one. |
+| `cache_table_name` | `"translation_diff_translations"` | Table `Stores::ActiveRecord` reads and writes. For a host with its own table-naming convention. See [SQL cache](sql-cache.md). |
+| `rate_limit_table_name` | `"translation_diff_rate_limits"` | Table `RateLimiters::ActiveRecord` reads and writes. As above. |
+| `active_record_base` | `nil` (`::ActiveRecord::Base`) | The class `Stores::ActiveRecord` and `RateLimiters::ActiveRecord` build their model from -- point this at a second database. It does not exempt this store from a Rails application's own read-replica routing; see [Rails replica routing](sql-cache.md#rails-replica-routing). See [SQL cache](sql-cache.md#active_record_base-a-second-database). |
+| `cache_prune_probability` | `0.0` | Chance, per write, that `Stores::ActiveRecord` prunes expired rows before returning. `0.0` is off, and a value outside `0.0..1.0` is refused at `configure` time; `rake translation_diff:prune` is the other way to prune. See [SQL cache](sql-cache.md#pruning-three-answers-none-imposed). |
+| `redis_url` | `ENV["REDIS_URL"]` | Where to connect for the Redis-backed cache store and rate limiter. Setting this is what makes `cache` default to `:redis` instead of `:memory`. |
+| `redis_pool_size` | `5` | Size of the connection pool built from `redis_url`. |
+| `redis_pool_timeout` | `5` | Seconds to wait for a connection from that pool before raising. |
+| `rate_limit` | `nil` | Character threshold per `rate_interval`. Unset with `rate_limiter` also unset means no rate limiting at all. Unset with `rate_limiter` set turns rate limiting on anyway, at that limiter's own default threshold -- 8,000 characters per `rate_interval`, the same default for both shipped limiters -- rather than the threshold you never set. |
+| `rate_interval` | `60` | Seconds over which `rate_limit` (or a limiter's own default threshold) is measured. **Actually enforced over roughly 5-600 seconds** -- see [The rate limiter contract](contracts.md#the-rate-limiter-contract). |
+| `rate_limiter` | `nil` | A registered name (`:redis`, `:active_record`) or an object satisfying the [rate limiter contract](contracts.md#the-rate-limiter-contract). `nil` with `rate_limit` also `nil` means no rate limiting; `nil` with `rate_limit` set resolves to `:redis`. Setting `rate_limiter` alone -- with `rate_limit` left unset -- is enough to turn rate limiting on, at the limiter's own default threshold; it no longer needs `rate_limit` set to avoid crashing. |
+| `segmenter` | `:pragmatic` | The sentence segmenter: a registered name or an object satisfying the [segmenter contract](contracts.md#the-segmenter-contract). |
+| `opaque_elements` | `%i[script style pre code]` | Element names `TranslationDiff::Passage` never treats as prose, whatever they contain, matched case-insensitively so `<PRE>` and `<STYLE>` count too. Read fresh on every passage rather than memoised, so a runtime change applies immediately, to the next translation -- and, since a call reads this from the configuration it is actually using, a context's own `opaque_elements` is honoured too, not the global value it was copied from. See [How it works](how-it-works.md#html). |
+| `instrumenter` | `nil` | Anything satisfying `ActiveSupport::Notifications`' `#instrument(name, payload) { }` interface. See [Instrumentation and logging](instrumentation.md). |
+| `logger` | `nil` | A standard `Logger` -- anything answering to `debug` and `warn` with a block. Receives one `debug` line per provider resolution, naming the provider class, and a `warn` line when a cache write fails; never content and never a credential. Note that `warn` must be a public method: a bare object inherits a private `Kernel#warn` and would raise instead of logging. See [Instrumentation and logging](instrumentation.md). |
+| `open_timeout` | `5` | Seconds an HTTP-backed provider waits to open a connection before raising `TranslationDiff::TransportError`. |
+| `timeout` | `30` | Seconds an HTTP-backed provider waits for a response before raising `TranslationDiff::TransportError`. |
+| `max_retries` | `3` | Retries `faraday-retry` attempts on a transport failure or a `429`/`500`/`502`/`503`/`504` response, with exponential backoff. `faraday-retry` honours a `Retry-After` header itself, so a `429` usually exhausts its retries before `TranslationDiff::RateLimitError` is ever raised. |
+| `validate_languages` | `true` | Whether `translate` refuses a source/target pair the shipped data doesn't list, before making a request. See [Languages](languages.md). |
+
+## Provider options
+
+Every provider declares its own configuration options, registered the moment
+`translation_diff` is required:
+
+| Provider | Options | Meaning |
+| --- | --- | --- |
+| `:deepl` | `deepl_api_key` (required) | Sent as `DeepL-Auth-Key`. Falls back to `ENV["DEEPL_AUTH_KEY"]`. |
+| | `deepl_api_base` | Overrides the automatic free/paid host selection (from the `:fx` suffix on the key). Rarely needed. |
+| `:google` | `google_api_key` (required) | Sent as the `key` query parameter. Falls back to `ENV["TRANSLATE_KEY"]`, then `ENV["GOOGLE_CLOUD_KEY"]`. |
+| | `google_project_id` | Declared for a future credentials path; not currently read -- an API key needs no project. Falls back to `ENV["TRANSLATE_PROJECT"]`. |
+| | `google_api_base` | Overrides the default `https://translation.googleapis.com`. |
+| `:azure` | `azure_api_key` (required) | Sent as `Ocp-Apim-Subscription-Key`. |
+| | `azure_region` | Sent as `Ocp-Apim-Subscription-Region`. Required by a multi-service Azure resource; a single-service resource needs no region. |
+| | `azure_api_base` | Overrides the default `https://api.cognitive.microsofttranslator.com`. |
+| `:modernmt` | `modernmt_api_key` (required) | Sent as `MMT-ApiKey`. |
+| | `modernmt_api_base` | Overrides the default `https://api.modernmt.com`. |
+| `:libretranslate` | `libretranslate_api_base` (required) | Every instance is self-hosted; there is no default to fall back to. |
+| | `libretranslate_api_key` | Sent as `api_key` in the request body. Most instances do not require one. |
+| `:amazon` | `amazon_access_key_id`, `amazon_secret_access_key`, `amazon_region` (all required) | Used to sign each request with `aws-sigv4`. No environment fallback: this library does not implement the AWS credential chain, so `AWS_ACCESS_KEY_ID` and friends are not read. |
+| | `amazon_session_token` | For temporary credentials. |
+| | `amazon_api_base` | Overrides the default `https://translate.<amazon_region>.amazonaws.com`. |
+
+A provider you register yourself can declare its own options the same way --
+see [Writing a provider](providers.md#writing-a-provider) below.
+
+## Changing configuration at runtime
+
+`provider`, `cache`, `segmenter` and `rate_limiter` each resolve to a
+collaborator on first use, and that collaborator is memoised. Writing an
+option afterwards rebuilds only the memoised collaborator(s) that option
+actually feeds, not the whole configuration:
+
+- `provider`, any option a provider declares for itself (`deepl_api_key`
+  and the like), and the timeouts (`open_timeout`, `timeout`,
+  `max_retries`) rebuild the provider. An HTTP-backed provider memoises a
+  Faraday connection built from the three timeouts, so a change to any of
+  them has to reach the provider or it never reaches the connection.
+- The cache options (`cache`, `cache_ttl`, `cache_max_size`,
+  `cache_table_name`, `active_record_base`, `cache_prune_probability`,
+  `cache_namespace`) rebuild the cache store.
+- `redis_url`, `redis_pool_size` and `redis_pool_timeout` rebuild the
+  connection pool and everything holding it -- the cache store and the rate
+  limiter both.
+- The rate options (`rate_limit`, `rate_interval`, `rate_limiter`,
+  `rate_limit_table_name`), `cache_namespace` and `active_record_base`
+  rebuild the rate limiter.
+- `segmenter` rebuilds the segmenter.
+- `logger` and `instrumenter` rebuild nothing -- both are read live, on
+  every use, and nothing memoised reads either one.
+
+Before this, nothing was ever rebuilt: an application wanting to switch
+`provider` at runtime had no way to do it short of `TranslationDiff.reset!`
+and reconfiguring from scratch, which also threw away a Redis pool, and
+everything built from it, that had no reason to go.
+
+Two options carry further than their names suggest, and both rebuild the
+rate limiter as well as the store. `cache_namespace` names the limiter's own
+bookkeeping namespace, so changing it at runtime moves the limiter too --
+it counts under the new namespace from the next check on, rather than
+continuing silently under the old one. `active_record_base` is the class the
+SQL-backed limiter builds its model from just as the SQL-backed store does,
+so pointing it at another database moves both.
+
+**A write that leaves an option at the value it already holds rebuilds
+nothing.** Only a value that actually changes invalidates a memoised
+collaborator, even one the option is declared to invalidate. This is what
+makes a per-request `TranslationDiff.configure { |c| c.cache_namespace =
+current_tenant }` safe: writing the same tenant on every request no longer
+rebuilds the cache store on every request. Writing a genuinely *different*
+value still rebuilds the store exactly as before, though, and if that store
+is the default `Stores::Memory`, a rebuilt store is a fresh, empty Hash --
+its contents are gone, and whatever it held has to be paid for again at the
+provider.
+
+`TranslationDiff.context` -- or `config.copy`, which it is built on -- is
+still the way to get a configuration that resolves everything afresh from
+its own values, independently of whatever the configuration it was copied
+from has already built.
+
+## Choosing the cache store
+
+`cache` unset means "choose for me": `Stores::Redis` when `redis_url` is
+configured, `Stores::Memory` otherwise, so the library works before any
+infrastructure does. Set `cache` explicitly (`:redis`, `:memory`, or your own
+object) to override that choice.
+
+## Contexts
+
+`TranslationDiff.context` returns a `TranslationDiff::Context`: an isolated
+configuration scope with the same `#translate` and `#preview` entry points
+as the `TranslationDiff` module itself, for multi-tenant applications and
+per-request overrides. It starts from a copy of the global configuration, so
+it inherits every value already set, and changes made inside it never touch
+the global configuration:
+
+```ruby
+tenant = TranslationDiff.context { |config| config.deepl_api_key = tenant_key }
+tenant.translate("Hello.", from: "en", to: "ru")
+
+TranslationDiff.config.deepl_api_key # unchanged
+```
+
+**The block is required.** `TranslationDiff.context` yields the copy for you
+to configure and there is nothing useful to hand back without one -- a copy
+nobody configured is just the global configuration. Called without a block it
+raises `LocalJumpError`.
+
+**A context starts with a cold cache.** Copying a configuration carries its
+option *values* over, but deliberately not the collaborators already built
+from them -- each context resolves its own provider, cache store, segmenter
+and rate limiter from its own values, independently of whatever the
+configuration it was copied from had already built. When `cache` is left
+unset, that resolves to `Stores::Memory`, an in-process store, so a freshly
+built context's store starts empty every time -- a short-lived, per-request
+context therefore caches nothing across requests. Configure `redis_url` (or
+assign one shared cache object explicitly) if contexts need to share a
+cache.
+
+**An object assigned directly is shared; a name is not.** Assign
+`config.cache`, `config.provider`, `config.segmenter` or `config.rate_limiter`
+a symbol and every context builds its own instance from it. Assign an
+*object* instead and that exact object -- the same connection pool, the same
+client -- is carried into every context copied from that configuration:
+someone who hands this library one connection pool means one connection
+pool.

data/docs/contracts.md

@@ -0,0 +1,187 @@
+# Contracts
+
+## The rate limiter contract
+
+Like `provider`, `cache` and `segmenter`, `rate_limiter` resolves a symbol
+through its own registry, `TranslationDiff::RateLimiters` -- `:redis` and
+`:active_record` are registered there. `config.rate_limiter_instance` is:
+
+- the object assigned to `config.rate_limiter`, if any -- an object still
+  bypasses the registry entirely, the same way it does for `cache`;
+- otherwise `nil` if both `rate_limiter` and `rate_limit` were never set --
+  and `Dispatcher#throttle` checks for that `nil` and skips rate limiting
+  entirely, so the common case costs nothing;
+- otherwise the registered limiter named by `config.rate_limiter`, or
+  `TranslationDiff::RateLimiters::Redis` when `rate_limiter` is left unset but
+  `rate_limit` is set -- built from `rate_interval`, `cache_namespace`, and
+  either `redis_url` (`:redis`) or `active_record_base` and
+  `rate_limit_table_name` (`:active_record`; see [SQL cache](sql-cache.md)).
+  `rate_limit` supplies the threshold when it is set; left unset, the
+  limiter falls back to its own default -- 8,000 characters per
+  `rate_interval` for both shipped limiters -- instead of crashing, so
+  setting `rate_limiter` alone is enough to turn a limiter on.
+
+An object assigned to `rate_limiter` must implement:
+
+```ruby
+# Called with the number of characters about to be sent to the provider.
+# Raises when the caller-defined threshold is exceeded.
+def check(size); end
+```
+
+Both shipped limiters raise `TranslationDiff::RateLimitExceeded` when the
+threshold is exceeded within the interval -- one class whichever limiter is
+configured, so switching from `:redis` to `:active_record` does not quietly
+stop a `rescue` from matching. They raise with a
+message naming the namespace, the threshold and the interval that were hit
+(`"rate limit reached for translation-diff: 8000 characters per 60
+seconds"`) -- never the text that tripped it. Neither `redis`
+nor `connection_pool` nor `ratelimit` is a dependency of this gem:
+`ratelimit` is required on the first check, so an application that
+configures no `rate_limit` never needs it, and its absence raises
+`TranslationDiff::Error` naming the gem to add. `activerecord` is never a
+dependency either -- see [SQL cache](sql-cache.md#the-activerecord-version-floor).
+
+**Upgrading to translation_diff 1.0.0: re-validate your `rate_limit` threshold.** Before this
+release, `RateLimiters::Redis` never actually limited anything -- a signature
+mismatch with the `ratelimit` gem meant it recorded hits under a subject
+`exceeded?` never read, so the threshold could never be reached. That bug
+shipped in every release since `v1.0.2` (2023-02-16). If you have
+`rate_limit` configured, your traffic has never actually been throttled by
+it; upgrading makes the limiter fire for the first time, against a value you
+may have set once and never seen exercised. Re-check that the threshold
+still reflects the traffic you actually want to allow before you upgrade.
+
+**`rate_interval` is silently clamped to roughly 5-600 seconds.** The
+built-in limiter constructs `Ratelimit.new` with no bucket options, so the
+gem's own fixed bucket span applies regardless of what you configure --
+measured: `rate_interval: 3600` behaves as `600`, and `rate_interval: 1`
+behaves as `5`. Combined with the fix above, an interval configured above
+600 seconds is now enforced over 600 seconds instead, which trips the
+limiter up to six times more eagerly than the configured value suggests.
+Keep `rate_interval` within 5-600 seconds if you want the configured number
+to be the enforced one.
+
+Both the clamp above and the upgrade note before it are about
+`RateLimiters::Redis`, which delegates its bucketing to the `ratelimit` gem.
+`RateLimiters::ActiveRecord` owns its own bucketing instead, and its window is
+sliding rather than tumbling: buckets are `rate_interval / 12` seconds wide
+(floored at 1 second), and a check sums every bucket touching the trailing
+`rate_interval` seconds -- including the oldest one, which is only ever
+partially inside that window, summed in full rather than pro-rated. So the
+window actually enforced is `rate_interval` to `rate_interval +
+rate_interval / 12` seconds: slightly stricter than configured, never
+looser, and with no external clamp. See
+[SQL cache](sql-cache.md#the-rate-limiter).
+
+## The segmenter contract
+
+`config.segmenter` decides where a text node is cut into sentence-sized
+cache units, the same way `config.provider` decides how a sentence gets
+translated. It defaults to `:pragmatic` and can be swapped for `:simple` or
+for any object implementing:
+
+```ruby
+# Returns the offsets at which a new sentence begins, always starting with 0
+# and strictly increasing. Slicing the source between consecutive offsets, and
+# from the last offset to the end, reconstructs the source exactly -- a wrong
+# boundary never corrupts the document, it only changes how the text is
+# grouped into cache units.
+#
+# language: is an ISO 639-1 code such as "en" or "ru" when the caller already
+# knows the source language, and nil when it does not -- see below.
+def split_offsets(text, language: nil); end
+```
+
+Two segmenters ship with this gem:
+
+- **`TranslationDiff::Segmenters::Pragmatic`** (the default) wraps the
+  [`pragmatic_segmenter`](https://github.com/diasks2/pragmatic_segmenter) gem,
+  which ships per-language rule sets rather than one rule set applied to every
+  script. Measured against the Golden Rules corpus, the de-facto benchmark for
+  sentence segmentation -- the `context "Golden Rules" do` block of each of
+  the 10 per-language spec files on `diasks2/pragmatic_segmenter`, 80
+  exemplars in total; a sample of the same corpus is in
+  `test/translation_diff/golden_rules_test.rb` -- it scores 76/80 against
+  `Simple`'s 47/80, and the gap is largest on languages that have no letter
+  case at all -- Arabic, Hindi, Armenian, Greek -- which `Simple` cannot
+  reason about by design.
+
+  Of the 4 exemplars `Pragmatic` misses, 3 are not boundary disagreements at
+  all: `pragmatic_segmenter`'s own expected value rewrites an incidental
+  newline into a space before comparing --
+
+      "This is a sentence\ncut off in the middle because pdf."
+        expected ["This is a sentence cut off in the middle because pdf."]
+        ours     ["This is a sentence\ncut off in the middle because pdf."]
+
+      "It was a cold \nnight in the city."
+        expected ["It was a cold night in the city."]
+        ours     ["It was a cold \nnight in the city."]
+
+  -- and the same shape recurs once in Japanese (`"これは父の\n家です。"`, expected
+  with the newline gone). In all three, `Pragmatic` finds exactly one
+  sentence, agrees on where it ends, and is scored wrong only because it
+  will not rewrite the source to match. Rewriting the source is exactly what
+  this gem's reconstruction invariant forbids, so this is a deliberate
+  choice, not a defect the score is hiding. The 1 remaining miss is a real
+  boundary disagreement, in English -- see the shadowing paragraph below.
+
+  Before segmenting, `Pragmatic` replaces every single newline (one with no
+  adjoining newline) with a space in a shadow copy of the text, segments the
+  shadow, and slices the *original* text at the recovered offsets --
+  `pragmatic_segmenter` otherwise treats almost any single newline as a
+  sentence boundary candidate even with no punctuation at all, which is a
+  false split (the harmful kind) on the incidental newlines that HTML text
+  nodes routinely carry from source formatting. A run of two or more
+  newlines (a real paragraph break) is left alone. This costs one Golden
+  Rules point (77 -> 76): one exemplar shaped like a bare list of items
+  separated by single newlines, with no punctuation, now segments as one
+  unit instead of three. That shape does not arise in this gem's actual
+  input -- HTML list items are separated by markup into distinct text nodes
+  already -- so the point is a deliberate trade, not a regression to chase.
+
+  Language codes are normalised before reaching `pragmatic_segmenter`:
+  downcased, with any region subtag after `-` or `_` dropped, and checked
+  against the codes `pragmatic_segmenter` actually has rules for, falling
+  back to English otherwise. DeepL -- this gem's own flagship provider --
+  sends codes exactly like `"RU"` and `"EN-GB"`; without normalising,
+  `pragmatic_segmenter`'s own lookup is case-sensitive and region-blind, so
+  those would silently miss their rule set entirely.
+- **`TranslationDiff::Segmenters::Simple`** is a zero-dependency, in-house
+  segmenter. It splits conservatively on punctuation followed by whitespace,
+  guarded by a handful of signals (a known abbreviation, an initial, digits on
+  both sides, a URL or email, or a lowercase letter immediately following --
+  the guard that gives it away as built for cased scripts). Reach for it if
+  you want no extra dependency and you only ever translate from languages
+  written in a cased script (Latin, Cyrillic, Greek's own script aside,
+  Armenian, and similar).
+
+Passing `from:` to `::translate` does more than skip a detection call (see
+[How it works](how-it-works.md) below): it is also the only way a segmenter sees
+the source language. When `from:` is omitted, the language is genuinely
+unknown at the time the text is segmented -- language detection needs the
+segmented text to build its sample, so segmentation cannot wait for it -- and
+`Pragmatic` falls back to English rules, which can mis-segment other
+languages (Russian abbreviations, for one). `Simple` ignores the argument
+entirely; its rules are language-neutral.
+
+`pragmatic_segmenter`'s cleaner rewrites the sentences it hands back in ways
+shadowing does not cover -- it collapses runs of three or more spaces, and it
+respaces abbreviations like `"Ph.D."` into `"Ph. D."`, among other things --
+so the sentence `Pragmatic` gets back does not always appear verbatim in the
+source any more. `Pragmatic` never guesses at an offset it cannot verify: it
+walks the returned sentences in order, keeps every offset it locates, and
+stops at the first one it cannot -- but the boundary at the end of the last
+sentence it did locate is not thrown away with the rest, since it was
+matched character for character too. Only the genuinely unrecoverable
+remainder is coarsened into one final unit; the verified prefix before it is
+still sliced off. This is a *coarsening*, not a failure -- the text still
+translates correctly, the cache unit is just larger than it could have
+been -- and it is silent by design,
+the same way a segmenter simply not splitting a node has always been
+acceptable. `TranslationDiff::Segmenters::Pragmatic::Error` (a
+`TranslationDiff::Error`) still exists and is still raised, but only if
+`Pragmatic` itself computes offsets that violate its own postcondition
+(starting at 0, strictly increasing, all within the text) -- not by ordinary
+use of `pragmatic_segmenter`, however it rewrites a sentence.

data/docs/development.md

@@ -0,0 +1,34 @@
+# Former name, upgrading, and development
+
+## Former name and upgrading
+
+This gem was published as `deepl_diff` through 2.2.0. `deepl_diff` is
+deprecated in favor of `translation_diff`, which is functionally the same
+gem under a name that no longer implies a dependency on DeepL specifically.
+
+**Upgrading from `deepl_diff`:** every cache key changed in translation_diff 1.0.0 -- the
+provider, the provider options and normalised language codes are now part of
+the key. Nothing cached previously is reused; the next translation of every
+sentence is a cache miss, once, everywhere. `TranslationDiff.api`, `.cache_store`,
+`.segmenter` and `.rate_limiter` -- the four module-level accessors earlier
+versions configured directly -- are gone; configure `TranslationDiff.config`
+(or use `TranslationDiff.configure`) instead. If you have `rate_limit`
+configured, also read the upgrading note in
+[The rate limiter contract](contracts.md#the-rate-limiter-contract): the limiter was
+never actually enforcing your threshold before translation_diff 1.0.0, and it starts doing so
+now. See [CHANGELOG.md](../CHANGELOG.md) for the full list of breaking changes.
+
+**If you registered a custom provider,** it must now subclass
+`TranslationDiff::Provider` (or `TranslationDiff::HTTPProvider`), declare
+`self.capabilities`, and implement `#translate(request)` taking a
+`TranslationDiff::Translation::Request` and returning a
+`TranslationDiff::Translation::Response` -- the duck-typed
+`#translate(texts, from:, to:, **options)` plus `#max_request_size` and
+`#max_batch_size` methods are no longer read at all. See [Writing a
+provider](providers.md#writing-a-provider).
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).

data/docs/errors.md

@@ -0,0 +1,92 @@
+# Errors
+
+Every error this gem raises inherits from `TranslationDiff::Error < StandardError`,
+so rescuing the gem's failures in one clause is a single `rescue TranslationDiff::Error`:
+
+```
+TranslationDiff::Error
+├── TranslationDiff::ConfigurationError         # a provider is missing a required option
+├── TranslationDiff::ProviderError              # the service answered and said no
+│   ├── AuthenticationError                     # 401/403
+│   ├── RateLimitError                          # 429, once faraday-retry's own retries
+│   │                                            # are exhausted -- carries #retry_after
+│   │                                            # when the service sent one
+│   ├── QuotaExceededError                      # 456
+│   ├── InvalidRequestError                     # any other 4xx
+│   └── ServiceError                            # 5xx, or anything else
+├── TranslationDiff::TransportError             # nobody answered: connection failed,
+│                                                # timed out, or TLS failed
+├── TranslationDiff::ResponseError              # the answer was well-formed HTTP but broke
+│                                                # this library's contract -- a body that
+│                                                # is not JSON, a provider that returned
+│                                                # the wrong number of translations, or one
+│                                                # that returned no translation for an input
+├── TranslationDiff::InvalidProviderError       # a class registered without inheriting
+│                                                # TranslationDiff::Provider, or any
+│                                                # provider -- registered or assigned --
+│                                                # whose cache_key is blank
+├── TranslationDiff::UnsupportedLanguageError   # the shipped language data doesn't list
+│                                                # this source/target pair for this
+│                                                # provider -- see docs/languages.md
+├── TranslationDiff::Translator::Error          # from: missing and the provider cannot
+│                                                # detect
+├── TranslationDiff::Previewer::Error           # the same, asked of a preview: detecting
+│                                                # a language is a paid request, and a
+│                                                # preview never makes one
+├── TranslationDiff::SentenceCache::Error       # provider options have no stable
+│                                                # serialisation for the cache key
+├── TranslationDiff::Batch::Error               # one sentence, once escaped, is larger
+│                                                # than the provider's declared limit
+├── TranslationDiff::Segmenters::Pragmatic::Error
+│                                                # Pragmatic computed offsets that
+│                                                # violate its own postcondition --
+│                                                # not raised by ordinary use
+└── TranslationDiff::RateLimitExceeded           # the configured rate_limit was exceeded --
+                                                 # one class whichever limiter noticed, so
+                                                 # switching `rate_limiter` between `:redis`
+                                                 # and `:active_record` cannot quietly stop a
+                                                 # rescue from matching
+```
+
+`RateLimitExceeded` carries a message naming the namespace,
+the threshold and the interval that were exceeded (`"rate limit reached for
+translation-diff: 8000 characters per 60 seconds"`) -- never the text that
+tripped it.
+
+Both SQL-backed collaborators report a database failure the same way. A
+cache write that the database refuses -- including under Rails'
+`prevent_writes` (a read-replica request, see
+[Rails replica routing](sql-cache.md#rails-replica-routing)) -- is rescued,
+redacted and swallowed, and the translation is returned anyway. The rate
+limiter's own write raises a redacted `TranslationDiff::Error` instead of
+continuing, because it runs before the provider does and a limiter that
+cannot count is not a limiter. Either way `rescue TranslationDiff::Error`
+around `translate` catches what a caller can catch, and no raw
+`ActiveRecord::ReadOnlyError` reaches it.
+
+`ProviderError` and its subclasses carry `#provider` (the registered name)
+and `#status` (the HTTP status code), so a caller can log or branch on which
+service and which response caused the failure without parsing the message.
+
+`TranslationDiff::Registry` -- which backs the provider, cache store and
+segmenter registries -- raises `TranslationDiff::Error` directly (not a
+dedicated subclass) for an unknown name, listing what is actually
+registered. `TranslationDiff::Batch::Error` is its own class rather than a
+direct `TranslationDiff::Error`, so a caller can catch "this sentence is too
+long for this provider" without also catching an unrelated registry miss; it
+is raised when one sentence is larger once escaped than the provider's
+declared `max_request_size` or `max_text_size` and so could never be sent
+even in a batch of its own. The message names a short prefix of the
+offending text and both numbers.
+
+`ArgumentError`, not a `TranslationDiff::Error`, is what
+`TranslationDiff.translate`, `TranslationDiff.preview` and their `Context`
+counterparts raise when `to:` is missing or `nil`. It is a caller's mistake
+before it is a translation, and the message names the keyword.
+
+**Renamed in translation_diff 1.0.0.** `TranslationDiff::Request::Error` is now
+`TranslationDiff::Translator::Error` and `TranslationDiff::Cache::Error` is
+now `TranslationDiff::SentenceCache::Error`; both classes they hung off are
+gone. `TranslationDiff::Chunker::Error` is gone with no replacement -- the
+condition it named now raises `TranslationDiff::Batch::Error`. A
+`rescue TranslationDiff::Error` catches all three exactly as before.