end_point_blank 0.6.0 → 0.12.0

data/README.md

@@ -8,8 +8,8 @@ auto-loads (railtie + middleware) when Rails is present.
 ## Capabilities
 
 - **Endpoint tracking** — every request/response passing through the Rack middleware is reported.
-- **Authorization** — outbound calls to other EndPointBlank-protected services are signed
-  (`Basic` client-credential or cached `Bearer` token), and inbound requests can be authorized
+- **Authorization** — outbound calls to other EndPointBlank-protected services carry a
+  `Bearer` access token (never this service's own client credentials), and inbound requests can be authorized
   against the EndPointBlank service before your action runs.
 - **Error, request, response, and log reporting** — background, queued, non-blocking delivery to
   the EndPointBlank intake API.
@@ -61,29 +61,51 @@ EndPointBlank::Writers::LogWriter.info("service started", { pid: Process.pid })
 
 ## Configuration
 
-`EndPointBlank.configure { |c| ... }` yields the `EndPointBlank::Configuration` singleton.
+`EndPointBlank.configure { |c| ... }` yields the `EndPointBlank::Configuration` settings and
+applies every assignment made inside the block together, only once the block returns without
+raising -- a block that raises leaves the configuration exactly as it was before the call.
 Every setting listed below can be set explicitly in that block, and most also fall back to an
 `ENDPOINTBLANK_*` environment variable, then to a built-in default.
 
+`c` is only valid for the duration of the block: once `configure` returns, whether the block
+returned normally or raised, `c` is frozen, and every String, Array or Hash value it holds is
+first replaced with its own frozen deep copy. A write made through a reference to `c` kept past
+the block always raises `FrozenError` -- a reassignment (`saved.app_name = "x"`) because `c`
+itself is frozen, and an in-place edit (`saved.masking_rules << rule`, `saved.app_name << "x"`,
+editing a rule Hash in place) because the value it points to is frozen too, not just `c`. A read
+that bypasses `c` -- `EndPointBlank::Configuration.instance.app_name`, or `EndPointBlank.logger`
+right after `c.logger = ...` earlier in the same block -- still sees the value from before the
+`configure` call started, not what the block has set on `c` so far, until the block returns and
+the change is applied.
+
+Assigning a String, Array or Hash through `c` copies it rather than storing the object itself:
+after `c.masking_rules = rules`, mutating the `rules` array you passed in no longer affects the
+live configuration -- call `configure` again to apply a further change. `logger`, `mask_hook`
+and `version_finder` are not String/Array/Hash, so they are held by reference like any other
+object the caller hands in: mutating one through a retained `c` (`saved.logger.level = ...`)
+still reaches the live value, the same as mutating it through `EndPointBlank.logger` or
+`Configuration.instance.logger` directly would.
+
 **Precedence: explicit `configure` value > `ENDPOINTBLANK_*` environment variable > default.**
 
 | `configure` setting | Env var fallback | Default | Notes |
 |---|---|---|---|
-| `client_id` | `ENDPOINTBLANK_CLIENT_ID` | `nil` | Used to build the `Basic` authorization header. |
+| `client_id` | `ENDPOINTBLANK_CLIENT_ID` | `nil` | Authenticates this service to its own intake (`Basic`). Never sent to a provider. |
 | `client_secret` | `ENDPOINTBLANK_CLIENT_SECRET` | `nil` | Paired with `client_id`. |
 | `base_url` | `ENDPOINTBLANK_BASE_URL` | `https://in.endpointblank.com` | Base for access-token, authorize, and endpoint-update APIs. |
 | `log_base_url` | `ENDPOINTBLANK_LOG_BASE_URL` | `https://log.endpointblank.com` | Base for error/request/response/log reporting APIs. |
 | `app_name` | `ENDPOINTBLANK_APP_NAME` | `Rails.application.name.underscore` if Rails is defined, else `nil` | Identifies your app to EndPointBlank. |
 | `env_name` | `ENDPOINTBLANK_ENV` | `RACK_ENV`, then `APP_ENV`, then `Rails.env` if defined, else `"production"` (resolved per-request by `SessionConfiguration.env_name`, not read directly off `Configuration`) | The environment name reported with each request/response payload. |
 | `logger` | — | A `::Logger.new($stdout, level: ::Logger::INFO)`, or `Rails.logger` under Rails (set by the railtie) | Any object with `.debug`/`.info`/`.warn`/`.error`/`.fatal` works. |
-| `worker_count` | — | `4` | Currently unused by the delayed writer (which always spins up 2 threads); reserved. |
+| `worker_count` | — | `4` | Number of background threads draining the delayed writer's queue. Falls back to 2 when set to `nil`. |
 | `token_ttl` | — | `nil` | Optional TTL (seconds) requested when generating a `Bearer` access token. |
-| `cache_ttl` | — | `300` | TTL (seconds) for the authorization decision cache. |
+| `cache_ttl` | — | `300` | TTL, in whole seconds, for the authorization decision cache. **Omit it to get the 300-second default.** `0` disables the cache; any positive `Integer` is that many seconds. Anything else — an explicit `nil`, a negative number, or a non-`Integer` such as `"300"` or `3.5` — raises `ArgumentError` from the `c.cache_ttl = ...` assignment itself, at configure time, and leaves the previous value in place, so a bad value stops your app at boot instead of surfacing on the first authorized request. The cache is a plain in-memory Hash scoped to this process (no `Rails.cache`, Redis, or other shared store), so a Puma or Unicorn worker and every separate app instance each hold their own cache and their own view of `cache_ttl`. Changing `cache_ttl` at runtime takes effect immediately for already-cached entries, not just new ones: an entry is valid only while it is within *both* its original write-time expiry and the *currently configured* `cache_ttl` measured from when it was written, so raising `cache_ttl` never extends an entry already in the cache, and lowering it shortens one on its next read. `cache_ttl = 0` disables the cache — the moment a read or a store in a given process observes it disabled, *that process's* entire cache is cleared, not just the entry being looked up or written, and a store made while disabled inserts nothing there. This is per process, not fleet-wide: each worker or instance clears only its own cache, and only once it has itself observed `cache_ttl` disabled and then handled an `Authorized` request (or a direct cache call) while disabled — a worker that is idle, or hasn't yet picked up the new config, keeps serving whatever it already cached until it does. (A disable followed by a re-enable with no cache read or store in between flushes nothing, since nothing observed the disabled state.) |
 | `trust_proxy_headers` | — | `true` | Whether the per-request `scheme`/`host`/`port` report honors `X-Forwarded-Proto`/`-Host`/`-Port`. See [Reported base URL](#reported-base-url). |
 | `masking_rules` | — | `[]` | Ordered list of masking rule hashes — see [Data masking](#data-masking). |
 | `mask_hook` | — | `nil` | Optional `->(payload, record_type_string) { payload }` run after `masking_rules`. |
 | `version_finder` | — | `nil` | Optional `->(request) { "1" }` overriding `EndPointBlank::Commands::VersionFinder`'s default header/param/path detection. |
 | `application_version` | — | `nil` | Reserved for reporting your app's own version. |
+| `derive_base_url_from_client_id` | — | `false` | Derive the intake hostname from a slug-prefixed `client_id` when no `base_url` is set. See [Intake hostname from `client_id`](#intake-hostname-from-client_id). Only `true` or `false`; anything else raises `ArgumentError` at configure time. |
 
 Note: there is also a bare `environment` accessor on `Configuration`, but it is not read by any
 code path in this gem (the real per-request environment name is `env_name`, described above) — do
@@ -118,6 +140,42 @@ hostname on an internal port. `host` is caller-controlled either way (it has alw
 the `Host` header), and none of these three values is ever used as an identity or
 authorization key, so the worst case is a wrong *suggestion* that an admin has to approve.
 
+### Intake hostname from `client_id`
+
+Each organization's intake will answer at its own hostname,
+`https://<slug>.in.endpointblank.com`, and every new `client_id` starts with
+that slug and a dot (`acima-x7k2mq.ijXI+MVwmrC5xH/9ZuGiQlAbAyobTqMa`). With
+`c.derive_base_url_from_client_id = true`, the gem picks its intake in this
+order:
+
+1. `base_url`, or else `ENDPOINTBLANK_BASE_URL`, if either is set;
+2. else, if the `client_id` carries a slug prefix,
+   `https://<slug>.in.endpointblank.com`;
+3. else `https://in.endpointblank.com`.
+
+A `client_id` carries a slug prefix only when the part before its first `.`
+has the exact shape of an organization slug and something follows the dot
+(`EndPointBlank::Configuration.client_id_slug`). A credential issued before
+slugs, including one with a `.` in it such as `my.client`, keeps calling
+`https://in.endpointblank.com`.
+
+**This is off by default, and turns on by default in a later release, once
+DNS and TLS for `*.in.endpointblank.com` are live.** Until then those
+hostnames do not resolve in production, so leave it off unless EndPointBlank
+has told you otherwise. With it off, the base URL is `base_url`, else
+`ENDPOINTBLANK_BASE_URL`, else `https://in.endpointblank.com`, whatever the
+`client_id`.
+
+The logs hostname is not derived: `log_base_url`, else
+`ENDPOINTBLANK_LOG_BASE_URL`, else `https://log.endpointblank.com`, as before.
+
+Every call to intake also sends `x-epb-sdk: ruby/<version>`, so
+EndPointBlank can tell which SDK versions use a credential before it moves an
+organization to another intake. The minimum Ruby version for a move is the
+release that turns `derive_base_url_from_client_id` on by default, **not**
+this one: with the option at its default here, the gem keeps calling
+`https://in.endpointblank.com` after its organization has moved.
+
 ### `configure` block example
 
 ```ruby
@@ -150,27 +208,146 @@ export ENDPOINTBLANK_ENV=staging
 
 ### Authorization
 
-`EndPointBlank::Authorization.header(base_url = nil)` builds the outbound `Authorization` header
-used by the gem's own HTTP calls: a cached `Bearer` token covering `base_url` when one is
-available (via `EndPointBlank::AccessTokens`), otherwise `Basic` credentials built from
-`client_id` / `client_secret` -- which covers both giving no target and a token that could not
-be obtained.
+`EndPointBlank::Authorization.header(base_url)` builds the `Authorization` header for an outbound
+call to a provider. It is always a `Bearer` token covering `base_url` (via
+`EndPointBlank::AccessTokens`, minting one if none is cached). It **never** falls back to `Basic`:
+a client must never send its own `client_id` / `client_secret` to a provider or to the provider's
+intake. When no token can be obtained it raises `EndPointBlank::TokenUnavailableError` instead.
 
 ```ruby
-EndPointBlank::Authorization.header # => "Basic ..."
+# Pass the URL you are about to call, NOT a hostname. Its userinfo, query and
+# fragment are removed before the token request; they are never sent to
+# intake, logged, or kept on the error.
+url = "https://api.example.com/orders"
+
+begin
+  auth = EndPointBlank::Authorization.header(url) # => "Bearer ..."
+  Excon.post(url, headers: { "Authorization" => auth }, body: payload)
+rescue EndPointBlank::TokenUnavailableError => e
+  # No token, so the provider was never called. e.outcome / e.status /
+  # e.failure say why (see "Why a token could not be minted" below): retry,
+  # degrade, or fail your own request -- but do not send credentials instead.
+  Rails.logger.warn(e.message)
+  raise
+end
+```
+
+`base_url` is required. Until this release `header` with no argument returned `Basic` credentials; that
+form is gone, and `header(nil)` or `header("")` raises `ArgumentError`, as does a URL that cannot be
+parsed into an http or https URL with a host (nothing is sent, and the message does not repeat the URL). The SDK's own calls to its
+own intake (authorize, token minting, endpoint updates, the log/request/response writers) still
+authenticate with `Basic`, which is safe because intake already holds this service's credential;
+they use the internal `EndPointBlank::Authorization.intake_header`, which is not for outbound
+calls.
 
-# Pass the URL you are about to call, NOT a hostname.
-# Strip any query string or fragment first -- intake rejects both.
-EndPointBlank::Authorization.header("https://api.example.com/orders") # => "Bearer ..." if a token is cached
+`TokenUnavailableError` (a subclass of `EndPointBlank::Error`) carries `base_url` (the URL with its
+userinfo, query and fragment removed; the raw value is never kept), `failure` (the
+`EndPointBlank::AccessTokens::Failure` recorded for the mint, or `nil`), and the shortcuts
+`outcome` and `status`. Its message names the stripped URL and a fixed reason for the outcome
+-- never intake's response body, which stays on `failure.reason` -- for example:
+
+```
+Could not mint an EndPointBlank access token for https://api.example.com/orders:
+intake rejected this application's client credential (HTTP 401); retrying cannot help --
+re-issue the credential. EndPointBlank never sends this service's client_id/client_secret to a
+provider, so there is no Basic-auth fallback and the call must not be made without a token.
 ```
 
+| `outcome` | Reason in the message |
+|---|---|
+| `:credential_rejected` | `intake rejected this application's client credential (HTTP 401); retrying cannot help -- re-issue the credential` |
+| `:request_rejected` | `intake refused the token request (HTTP <status>); check the URL and that a grant covers the target` |
+| `:server_error` | `intake failed to issue a token (HTTP <status>); this may be transient` |
+| `:transport_error` | `intake could not be reached (timeout, connection refused or retries exhausted); this may be transient` |
+| `:transport_error`, the mint raised | `the token request failed unexpectedly` |
+| none recorded | `the token request failed for an unknown reason` |
+
+` (HTTP <status>)` is left out when there is no status.
+
+A mint that raises rather than reporting a failure -- a bug, not intake being unreachable -- is
+reported as this error too, with outcome `:transport_error`, `unexpected?` true and the exception
+as `cause`; its message is not copied into the error's. Only a missing credential's
+`ConfigurationError` escapes `header` as itself.
+
+The SDK's own calls to its intake raise `EndPointBlank::ConfigurationError` (also a subclass of
+`EndPointBlank::Error`) when `client_id` or `client_secret` is missing or empty, rather than
+sending an empty `Basic` credential.
+
 The argument is the URL you are about to call. intake matches it against registered base URLs by
 longest path prefix, so you need not know how the target registered itself -- `header` for
 `https://api.example.com/orders/42` reuses a token already cached for
 `https://api.example.com/orders`. `EndPointBlank::AccessTokens` caches one token per base URL
 intake resolves to, not one per process, so a service that calls several targets holds a token
-for each. A URL that does not match character-for-character (a different case, a query string,
-an unregistered path) simply misses and mints a new token -- it never guesses.
+for each. The lookup uses the URL with its userinfo, query and fragment removed and its scheme and
+host lowercased, as intake does; beyond that, a URL that does not match character-for-character
+(a different path case, an unregistered path) simply misses and mints a new token -- it never guesses.
+
+### Why a token could not be minted
+
+`EndPointBlank::AccessTokens.token` answers with a token String or `nil` (or raises, as itself,
+anything the mint raised that is not a transport error -- only `Authorization.header` wraps that),
+which is all most callers need. When `nil` is not enough — when you want to know whether retrying could possibly
+help — call `token_result` instead, which answers with the token or the `Failure` for that call:
+
+```ruby
+url = "https://api.example.com/orders"
+result = EndPointBlank::AccessTokens.token_result(url)
+
+if result.is_a?(EndPointBlank::AccessTokens::Failure)
+  failure = result
+
+  case failure.outcome
+  when :credential_rejected
+    # intake answered 401. Permanent until the credential itself changes:
+    # re-issue it in the portal and update client_id / client_secret.
+    raise "EndPointBlank credential rejected (#{failure.reason})"
+  when :request_rejected
+    # A 400 or 422: intake could not resolve the target or source
+    # application, or the request itself was malformed. Retrying will not fix
+    # it, but the credential is fine.
+  when :server_error, :transport_error
+    # A 5xx, an unusable response, a timeout, a refused connection. Try again.
+  end
+end
+```
+
+`EndPointBlank::AccessTokens.last_failure(url)` still answers the last failure recorded for a
+URL, and returns `nil` once a mint for that URL succeeds again, so it never reports a problem
+that has already cleared. It is a shared slot, though: read after `token` returned `nil`, another
+thread may already have cleared or replaced it. When you need the reason for your own call, use
+`token_result`, whose `Failure` is captured inside the cache's lock for that call. A `Failure` carries `base_url`, `outcome`, `status` (the HTTP
+status, or `nil` when no usable one was obtained), `reason`, and `at`, and answers
+`#credential_rejected?`, `#request_rejected?`, `#server_error?` and `#transport_error?`.
+
+`token`, `token_result`, `exists?` and `last_failure` all remove the URL's userinfo, query and
+fragment first, so the cache, the failure record (and its `base_url`) and the log lines only ever
+hold the stripped URL. A URL that cannot be parsed into an http or https URL with a host is never sent:
+`token_result` answers a `:request_rejected` `Failure` with no `status` and no `base_url`, and
+nothing is recorded.
+
+There is deliberately no `#retriable?` or other single retry/no-retry boolean. Retrying a `400`
+or a `422` is exactly as futile as retrying a `401` — intake answers `400` for an invalid
+`token_ttl` or a missing `base_url`, and `422` when the target or source application cannot be
+resolved — so a boolean would have to answer for cases whose only honest answer is "it depends
+what you are going to do about it". Branch on the outcome instead.
+
+Classification is on the HTTP status first and the body second. A `401` whose body will not parse
+is still `:credential_rejected`: the SDK reaches intake through a proxy, and a WAF or load
+balancer can answer 401 with an HTML page intake never generated. `:transport_error` means one
+thing only — no usable HTTP status was obtained because the request never completed (an Excon,
+socket, SSL or timeout error). Anything else raised while minting is not a transport error and
+propagates.
+
+The body decides exactly one thing, and only on a 2xx: whether a token was actually minted. A
+success means a token is there to read — the body parsed and carries a non-empty `token` and the
+non-empty `base_url` to cache it under. A 2xx that will not parse, or carries no token, or a
+token with no `base_url`, is a `:server_error` keeping its real 2xx status, because intake's
+`base_url` is `NOT NULL` and it answers a 4xx rather than minting when the URL resolves to
+nothing — so a 2xx missing one is a broken server, not a refused request.
+
+The same verdict is available one level down, without the cache, from
+`EndPointBlank::Commands::GenerateAccessToken.token_result(base_url)`, which returns an
+`AccessTokenResult` with `outcome`, `status`, `payload` and the same predicates.
 
 Under Rails, protect an inbound endpoint by including the `Authorized` concern in a controller —
 it calls `EndPointBlank::Commands::EndpointAuthorize.authorize(request)` before the action, and
@@ -197,6 +374,32 @@ preserve it) and you registered the external hostname in the portal, either upda
 registered hostname to the internal one the app now reports, or configure the proxy to preserve
 `Host`. Deployments where `Host` and `X-Forwarded-Host` agree are unaffected.
 
+The `Authenticated` concern is the lighter sibling: it calls
+`EndPointBlank::Commands::BasicAuthenticate.authenticate(request)` before the action and refuses
+the same way, but records nothing on the Rack env, sets no deprecation headers and — matching
+every other SDK's authenticate path — does not cache intake's answer.
+
+```ruby
+class OrdersController < ApplicationController
+  include EndPointBlank::Rails::Authenticated
+end
+```
+
+`EndPointBlank::UnauthorizedError#status` is intake's own verdict, from either concern, so
+`rescue_from` can render it directly. 401 and 403 are different remedies and must not be
+collapsed:
+
+| intake answered | `error.status` |
+| --- | --- |
+| 401 | `401` — re-check or re-issue the credential |
+| 403 | `403` — ask for a grant covering this endpoint |
+| any other non-201 | that status, verbatim |
+| nothing at all | `503` — the check could not be made, so nothing judged this caller |
+
+`UnauthorizedError.new(message)` still defaults to 401; the status is an optional second
+argument. Raise the instance (`raise UnauthorizedError.new(msg, status)`) rather than
+`raise UnauthorizedError, msg` — the two-argument form cannot carry a status.
+
 ### Error reporting
 
 Exceptions raised while `EndPointBlank::Middleware::Rack::ReportInteraction` is on the stack are
@@ -226,8 +429,21 @@ EndPointBlank::Writers::LogWriter.fatal("out of workers")
 
 All writers (`RequestWriter`, `ResponseWriter`, `ExceptionWriter`, `LogWriter`) enqueue their
 payload onto a bounded, in-memory queue (`DelayedWriter`, capacity 1000, drop-oldest under
-sustained backpressure) drained by two background threads that POST batches via `excon`. Delivery
-is fire-and-forget and never raises into your request cycle.
+sustained backpressure) drained by `worker_count` background threads that POST batches via `excon`,
+six payloads per request. Batches are cut by position, so two identical payloads are two payloads.
+Delivery is fire-and-forget and never raises into your request cycle.
+
+A worker thread does not die. A batch can be lost — the intake may be unreachable, or the send path
+may raise something nobody anticipated — but the loop catches every `StandardError`, logs it at
+`error` level through `EndPointBlank.logger` with a running count of consecutive failures, backs off
+(0.1s, doubling, capped at 30s), and keeps draining; the count resets on the first clean pass. Only
+an error outside `StandardError` — `SystemExit`, `Interrupt`, `SignalException`, `NoMemoryError`,
+i.e. the process itself going down — ends a worker.
+
+A writer may optionally define `on_success(response)` and `on_failure(response)` to hear about each
+batch. `on_failure` receives `nil` when the intake never answered at all: `Commands::Http` returns
+`nil` once its three attempts are exhausted, which is the absence of a status rather than a failing
+one. A writer that defines neither is unaffected.
 
 ### Data masking
 
@@ -290,6 +506,7 @@ end
 
 class OrdersController < ApplicationController
   include EndPointBlank::Rails::Authorized   # authorize inbound requests before each action
+  # or: include EndPointBlank::Rails::Authenticated  # authenticate only, no caching
   include EndPointBlank::Rails::Versioned
 
   version ["v1", "v2"], only: [:index]

data/end_point_blank.gemspec

@@ -8,9 +8,9 @@ Gem::Specification.new do |spec|
   spec.authors = ["Robert A. Lasch"]
   spec.email = ["rlasch@gmail.com"]
 
-  spec.summary = "Ruby/Rails client for EndPointBlank — endpoint tracking, authorization, and error/request/response/log reporting."
-  spec.description = "EndPointBlank client library for Ruby. A framework-agnostic core runs in plain Ruby / Sinatra, with Rails supported as an auto-loaded adapter. Provides API endpoint tracking, authorization, and error/request/response/log reporting."
-  spec.homepage = "https://github.com/EndPointBlank/end_point_blank_rails"
+  spec.summary = "Ruby and Rails SDK for EndPointBlank: authorize service-to-service API calls, report endpoint versions, and see which clients still call deprecated API versions."
+  spec.description = "Ruby and Rails SDK for EndPointBlank: authorize service-to-service (machine-to-machine) API calls, report endpoint versions, and see which clients still call deprecated API versions before you sunset them. A framework-agnostic core runs in plain Ruby / Sinatra, with Rails supported as an auto-loaded adapter."
+  spec.homepage = "https://endpointblank.com"
   spec.required_ruby_version = ">= 3.4.2"
   spec.license = "Nonstandard"
 
@@ -18,6 +18,8 @@ Gem::Specification.new do |spec|
 
   spec.metadata["homepage_uri"] = spec.homepage
   spec.metadata["source_code_uri"] = "https://github.com/EndPointBlank/end_point_blank_rails"
+  spec.metadata["documentation_uri"] = "https://endpointblank.com/docs/sdk-setup"
+  spec.metadata["bug_tracker_uri"] = "https://github.com/EndPointBlank/end_point_blank_rails/issues"
   spec.metadata["changelog_uri"] = "https://github.com/EndPointBlank/end_point_blank_rails/releases"
 
   # Specify which files should be added to the gem when it is released.