clicksend 1.1.0 → 1.2.0

data/README.md

@@ -64,7 +64,7 @@ message.message_id # => "1ABC3200-C38C-6308-BE4B-C7C51D01DCF0"
 | You want to… | Use |
 |---|---|
 | Send SMS from a Ruby app and track delivery and replies, with safe defaults, typed errors, instrumentation and a test fake | **this gem** |
-| Call the occasional ClickSend endpoint this gem doesn't wrap (price a message, cancel a scheduled one, list templates) with the same authentication, timeouts, retry rules and errors | **this gem's [`client.request`](#calling-other-clicksend-endpoints)** |
+| Call the occasional ClickSend endpoint this gem doesn't wrap (price a message, list templates, view statistics) with the same authentication, timeouts, retry rules and errors | **this gem's [`client.request`](#calling-other-clicksend-endpoints)** |
 | Work with large parts of the API (email, campaigns, contacts, numbers, automations, subaccounts, and so on) with generated models for each | ClickSend's official SDK, [`clicksend_client`](https://rubygems.org/gems/clicksend_client) |
 
 The two gems can be used side by side ([namespaces differ](#using-it-alongside-the-official-sdk)).
@@ -183,6 +183,37 @@ Unknown keywords raise `ArgumentError`, so typos can't be silently ignored.
 > ClickSend currently pauses SMS containing URLs for new customers until URL messaging is
 > approved ([docs](https://developers.clicksend.com/docs/messaging/sms)).
 
+### Cancelling a scheduled message
+
+> **Experimental in 1.2.** The successful answer (HTTP 200, `SUCCESS`) is covered by contract
+> tests against ClickSend's published API description, but has **not been observed live**.
+> ClickSend's free test number doesn't hold scheduled messages: in a live check (2026-10-06) a
+> message scheduled an hour ahead showed as `Completed` in history within seconds, so the test
+> number can't exercise a successful cancel. Cancelling those test-number messages answered
+> HTTP 404 `NOT_FOUND`, which the gem raises as `Clicksend::NotFoundError`. The API may change in a
+> minor release once a real cancellation has been observed.
+
+```ruby
+message = client.sms.deliver(to: user.phone, body: "Your appointment is tomorrow", schedule: Time.now + 86_400)
+client.sms.cancel(message.message_id) # => nil when ClickSend answers SUCCESS
+```
+
+ClickSend documents only the successful case. In the live check, cancelling a message that was no
+longer scheduled, cancelling the same ID twice and cancelling a random ID all answered the same
+404 `NOT_FOUND` ("Record not found."). So a `NotFoundError` doesn't tell you why: the ID may be
+unknown, or the message may no longer be cancellable, and it does **not** mean the message was
+sent. Nor does "no exception" prove the message won't go out. If it matters, for example before
+you schedule a replacement, check `client.sms.history(message_id: message.message_id)` for the
+status `"Cancelled"`.
+
+`cancel` is not retried after a timeout or 5xx, because ClickSend documents no idempotency for it.
+Such a failure raises an [`AmbiguousRequestError`](#when-a-sends-outcome-is-unknown): the message
+may or may not have been cancelled. Calling `cancel` again can't send anything, but may raise even
+if the first call worked; history tells you which.
+
+There is deliberately no wrapper for `PUT /v3/sms/cancel-all`: without a `custom_string` filter it
+cancels every scheduled message on the account.
+
 ### Many messages in one request
 
 ```ruby
@@ -215,7 +246,7 @@ accepted. The gem never retries such a send. It raises the error extended with
 
 ```ruby
 begin
-  client.sms.deliver(to: user.phone, body: text, custom_string: "otp:#{attempt.id}")
+  client.sms.deliver(to: user.phone, body: text, custom_string: "otp:#{otp.id}")
 rescue Clicksend::AmbiguousRequestError => e
   e.class          # => Clicksend::TimeoutError (or ServerError, ConnectionError, MalformedResponseError,
                    #    or any APIError ClickSend reported inside a 2xx answer)
@@ -231,17 +262,27 @@ message ClickSend refused) are not ambiguous. Apart from the refused connection,
 inference from ClickSend's documentation and observed behaviour, not a guarantee. An error ClickSend reports inside a 2xx
 answer is ambiguous whatever its code, because that behaviour is undocumented.
 
-To check, look the message up in [history](#message-history) by recipient and match your own
-`custom_string`:
+To check, search [history](#message-history) for your recipient and `custom_string`:
 
 ```ruby
-sent = client.sms.history(to: user.phone, date_from: started_at - 60)
-  .auto_paging_each.find { |record| record.custom_string == "otp:#{attempt.id}" }
+records = client.sms.search_history(to: user.phone, custom_string: "otp:#{otp.id}", sent_after: started_at)
+records.any?  # true: ClickSend accepted it (records.first.status says how far it got)
+              # false: nothing is known yet. This is NOT proof that it wasn't sent.
 ```
 
-ClickSend doesn't document how soon a sent message appears in history, so **a message missing
-from history is not proof that it wasn't sent**. Whether to resend is your decision: for a login
-code, letting the user request another is usually safer than resending automatically.
+`search_history` asks for the recipient's history (`q=to:`) from five minutes before
+`sent_after` (ClickSend doesn't say whether its date filters are inclusive or which clock they
+use), reads every page, and keeps only outbound rows whose `to` and `custom_string` are exactly
+yours: `custom_string` isn't a filter ClickSend offers, and it calls a filter value a "text or
+keyword", not an exact match. `to` must be E.164 (`"+61411111111"`), the form history stores.
+Each page of 100 rows is one request against ClickSend's undocumented rate limits.
+
+ClickSend doesn't document how soon a sent message appears in history, keeps history for about
+four months, and can de-identify recipients on request, so **an empty result is not proof that
+nothing was sent**. Whether to resend is your decision: for a login code, letting the user
+request another is usually safer than resending automatically. Use a `custom_string` that names
+the message you meant to send (here the OTP record), not one try at sending it: a job that runs
+again reuses it, so a match from an earlier try counts.
 
 ## Delivery receipts and replies
 
@@ -308,38 +349,68 @@ end
 works out which of the two it was given. Pass the body parameters, not the route ones, so your
 secret isn't kept in `#raw`.
 
-> **Experimental.** ClickSend doesn't document the push format, and no real push has been captured
-> for this gem yet: the field names come from ClickSend's polling API and archived docs. The
-> `Clicksend::Webhook` API may change in a minor release.
-
-**ClickSend documents no way to authenticate webhooks.** There is no documented signature, shared
-secret or HMAC. ClickSend's current documentation lists no source IP addresses; archived help pages
-(around 2019–2021, no longer published) listed some and said pushes come from a fixed pool. That
-list can't be checked against today's infrastructure, so this gem doesn't support or recommend IP
-allowlisting. Treat anyone who learns the URL as able to send a fake receipt. This gem therefore
-offers no "verify" method. Instead:
-- put an unguessable secret in the URL, compare it in constant time, and use HTTPS. A secret in the
-  path appears in access logs (Rails logs the path, and proxies and APM tools often keep it), so
-  restrict who can read them, and filter `body`, `from` and `to` with `filter_parameter_logging`;
+> **Experimental.** ClickSend's current docs don't define the push format, and no real push has
+> been captured for this gem yet. The field names come from ClickSend's polling API, which an
+> archived ClickSend help article and ClickSend's own n8n and Power Automate integrations agree
+> with. The `Clicksend::Webhook` API may change in a minor release.
+
+**ClickSend's current docs describe no way to authenticate webhooks**: no signature, HMAC or shared
+secret. An archived help article listed six source IP addresses, but it is no longer published and
+was last updated years ago, so don't build an allowlist on it. Treat anyone who learns the URL as
+able to send a fake receipt. This gem therefore offers no "verify" method. Instead:
+- put an unguessable secret in the URL, compare it in constant time, and use HTTPS (archived
+  ClickSend help says the certificate chain must be valid). A secret in the path appears in access
+  logs: Rails' request log shows the path as is (Rails 8.1 filters only query parameters, such as
+  `?token=`), and proxies and APM tools often keep the full URL. Restrict who can read them, and
+  add the personal fields (`body`, `message`, `from`, `to`, `sms`, `originalsenderid`,
+  `original_body`, `originalmessage`) to `filter_parameters`;
+- to rotate the secret, accept the old and the new one until every rule uses the new URL;
 - treat a push as a hint. A receipt can probably be confirmed with `client.sms.receipt(message_id)`
   (not yet verified for an account with only URL rules), at one API call per check. An
   inbound message can't be fetched by its ID through any wrapped or verified endpoint; the closest
   check is `client.sms.history(from: number)`. Be careful acting on unconfirmed replies such as
   "STOP";
-- handle pushes idempotently. Several rules can match, and (according to ClickSend's archived docs)
-  a non-200 answer is retried every 10 minutes, up to 10 times. Key inbound messages on
-  `message_id`. Key receipts on `message_id` **and** `status_code`: ClickSend's gateway codes include
-  states that aren't final (200, 300), so one message can legitimately produce more than one
-  receipt, and deduplicating on `message_id` alone could discard the final 201 or 301. When
-  receipts for a message disagree, prefer a final code;
-- answer 200 quickly and do the work in a job.
+- handle pushes idempotently. Several rules can match (up to 10 per inbound message, and
+  integrations such as Zapier create their own rules), and a non-200 or slow answer is retried:
+  ClickSend's archived pages say either every 10 minutes up to 10 times, or with backoff over hours.
+  An inbound `message_id` is unique. A sent message may get more than one receipt (the pending
+  codes 200 and 300 can change), so key receipts on `message_id` and `status_code`, and don't let a
+  pending receipt overwrite a final one (`delivered?` or `failed?`);
+- answer 200 quickly and do the work in a job. If the job parses or confirms the push, add
+  `discard_on Clicksend::Webhook::InvalidPayload` below any `retry_on Clicksend::Error`: a payload
+  that can't be parsed won't parse on a retry.
+
+```ruby
+# Two secrets during a rotation; ActiveSupport's secure_compare also handles different lengths.
+def valid_secret?(given)
+  secrets = Rails.application.credentials.clicksend_webhook_secrets # e.g. [new, old]
+  secrets.any? { |secret| ActiveSupport::SecurityUtils.secure_compare(given.to_s, secret) }
+end
+
+# Deduplicating receipts in a job: one row per message, final statuses win.
+# (Unique index on sms_deliveries.message_id.)
+def record(receipt)
+  delivery = SmsDelivery.create_or_find_by!(message_id: receipt.message_id)
+  return if delivery.final? && receipt.pending? # final?: your own check for status_code 201 or 301
+
+  delivery.update!(status_code: receipt.status_code, reported_at: receipt.reported_at)
+end
+```
 
 Inbound rules post form fields by default, or use a query string (`webhook_type: "get"`) or JSON
 (`"json"`); for JSON, pass `JSON.parse(request.raw_post)` or Rails' parsed body parameters. Receipt
-pushes are form-encoded according to ClickSend's archived docs. The parsers use the field names of
-the polling API, which match the archived push documentation, and reject payloads with too many
-fields, oversized or non-UTF-8 values, or nested values in the fields they read. Pass a plain
-Hash of the body parameters rather than Rails' `params`, which also holds route parameters.
+pushes are form-encoded according to ClickSend's archived help. According to the same sources,
+pushes also carry `user_id` and legacy duplicates (`message`, `sms`, `originalsenderid`,
+`messageid`, `customstring`, ...), which are kept in `#raw`, and a JSON push may send `timestamp`
+and `user_id` as numbers; either form is accepted. The parsers reject payloads with too many fields, oversized
+or non-UTF-8 values, or nested values in the fields they read. Pass a plain Hash of the body
+parameters rather than Rails' `params`, which also holds route parameters.
+
+Some things a receiver should expect, according to ClickSend's archived help: the same receipt
+format is used for voice, email and fax receipts (check `receipt.message_type` if those rules share
+your URL), and an inbound MMS arrives through the SMS inbound rules with its attachment as a link
+that expires after 7 days. To see a push without sending an SMS, the dashboard's **Add Test Reply**
+button on a rule posts an example to its URL.
 
 ## Message history
 
@@ -356,7 +427,7 @@ For this endpoint ClickSend documents a single `q=field:value` filter. Its gener
 documentation also describes several comma-separated fields with an `operator`, but not for history,
 and that hasn't been verified here. So `history` takes at most one of `to:`, `from:`, `status:` and
 `message_id:`, plus `date_from:`, `date_to:` and `order:` (`:asc` or `:desc`). `custom_string` isn't
-a documented filter; match it yourself.
+a documented filter; match it yourself, or use [`search_history`](#when-a-sends-outcome-is-unknown).
 
 ## Account balance
 
@@ -485,8 +556,14 @@ check whether that library retries requests by itself.
 
 ## Background jobs
 
-Job frameworks retry failed jobs, which can undo the gem's care about duplicates. Two rules keep a
-send job as safe as the gem can make it:
+Job runners retry failed jobs and re-run interrupted ones, and either can turn a send whose outcome
+is unknown into a second SMS. The measurements in this section come from experiments with
+ActiveJob 8.1.4, Sidekiq 8.1.7 and ActiveRecord 8.1.4 (SQLite) against local stand-ins for
+ClickSend, not from ClickSend itself.
+
+### A send job
+
+Two rules keep a send job as safe as the gem can make it:
 - **Never let an ambiguous send be retried**, and make sure nothing after it raises.
 - **Other failed sends may be retried.** The gem treats them as not processed: refused
   connections, 429s, 4xx responses and per-message rejections. For 4xx and rejections that is
@@ -502,12 +579,12 @@ class SendSmsJob < ApplicationJob
 
   def perform(notification_id)
     notification = Notification.find(notification_id)
-    CLICKSEND.sms.deliver(to: notification.phone, body: notification.text, custom_string: "notification:#{notification_id}")
+    CLICKSEND_JOBS.sms.deliver(to: notification.phone, body: notification.text, custom_string: "notification:#{notification_id}")
   rescue Clicksend::AmbiguousRequestError
-    # The message may have gone out. Hand over to a reconciliation step, and make sure nothing
-    # here raises: an exception would make the job runner retry the send.
+    # The message may have gone out. Hand over to reconciliation, and make sure nothing here
+    # raises: an exception would make the job runner retry the send.
     begin
-      ReconcileSmsJob.perform_later(notification_id) # e.g. checks sms.history for the reference
+      ReconcileSmsJob.perform_later(notification_id) # e.g. checks sms.search_history for the reference
     rescue => e
       Rails.logger.error("SMS for notification #{notification_id}: outcome unknown, reconciliation not enqueued (#{e.class})")
     end
@@ -515,22 +592,179 @@ class SendSmsJob < ApplicationJob
 end
 ```
 
-Ambiguous errors are rescued inside `perform`, so `retry_on` only sees errors the gem treats as
-not processed. When `retry_on` gives up, Rails re-raises the error and your queue backend may
-retry the job again.
+`CLICKSEND_JOBS` is a client with a shorter timeout, explained under
+[the timeout budget](#the-timeout-budget). Ambiguous errors are rescued inside `perform`, so
+`retry_on` only sees errors the gem treats as not processed.
+
+**Declaration order matters.** `retry_on` and `discard_on` declare `rescue_from` handlers, which
+Rails tries from the bottom up, a job's own before those it inherits. For a send that timed out
+after ClickSend had accepted it:
+
+| The job declares | Messages sent |
+|---|---|
+| only `retry_on Clicksend::Error` | 2 |
+| the recipe above | 1 |
+| `retry_on Clicksend::Error`, then `discard_on Clicksend::AmbiguousRequestError` | 1 |
+| the same two lines in the opposite order | **2** |
+| `discard_on Clicksend::AmbiguousRequestError`, under a `retry_on StandardError` inherited from `ApplicationJob` | 1 |
+
+`discard_on` accepts the `AmbiguousRequestError` module, so it works when it is declared last.
+But moving one line breaks it silently, and a discarded job reconciles nothing unless you give
+`discard_on` a block. Rescuing inside `perform` doesn't depend on order.
+
+**Retry layers stack.** When `retry_on` runs out of attempts it re-raises the error, and the queue
+backend then retries the job under its own policy (Sidekiq: 25 retries by default). Attempts
+multiply: with a persistent 429, `retry_on Clicksend::Error, attempts: 2` made 6 HTTP requests
+(2 job runs, each with the gem's 3 attempts) before the error reached the backend. On Sidekiq's
+ActiveJob adapter, a job that did *not* rescue an ambiguous error sent twice under
+`retry_on attempts: 2`, and was then in Sidekiq's retry set, ready to send a third time. Keep
+ambiguous errors inside `perform`, and pass `retry_on` a block if the backend shouldn't retry after
+it: with a block, `retry_on` calls it instead of re-raising.
+
+### The timeout budget
+
+When a job runner stops (a deploy, a scale-down), it waits a limited time for running jobs, then
+puts the unfinished ones back on the queue. A send still waiting for ClickSend's answer then runs
+again, and the gem never sees how the first attempt ended. A real Sidekiq 8.1.7 process stopped
+mid-send with `-t 2` re-queued the recipe's job, and the message was received **twice** with the
+gem's default 30s read timeout. With `timeout: 1`, the send timed out first, the recipe handled the
+ambiguous error, and the message was received once.
+
+So give jobs a client whose attempts end well inside the runner's shutdown timeout:
+
+```ruby
+# config/initializers/clicksend.rb
+CLICKSEND = Clicksend::Client.new(logger: Rails.logger, instrumenter: ActiveSupport::Notifications)
+# Sidekiq waits 25s by default (-t, :timeout). One attempt here gives up after about 5s without a
+# connection or 15s without data from ClickSend (each read gets 15s, so an answer that trickles in
+# can take longer), and max_retries: 0 leaves not-processed failures to retry_on.
+CLICKSEND_JOBS = CLICKSEND.with(timeout: 15, max_retries: 0)
+```
+
+- The window that matters runs from sending the request to reading the answer. A worker stopped
+  while connecting or backing off has sent nothing, so its re-run sends once. Retries inside the
+  gem add attempts, each with its own window: keep their total inside the budget too, or turn
+  them off as above.
+- Sidekiq's 25 seconds comes from its source (8.1.7), and the duplicate was measured with Sidekiq.
+  Other limits were not tested here: Solid Queue's README gives a default `shutdown_timeout` of
+  5 seconds; Kubernetes (`terminationGracePeriodSeconds`) and Heroku limit how long a stopping
+  process may run; queues with a visibility timeout redeliver a job that runs longer than it.
+  Check yours, and raise it rather than cutting the client's timeout below what ClickSend needs.
+- A shorter timeout turns more slow sends into ambiguous ones: more reconciliation, never a
+  duplicate.
 
 **Never wrap a send in `Timeout.timeout`.** It interrupts the thread at an arbitrary point, which can
 be after ClickSend has already received the message, and raises a plain `Timeout::Error`. That isn't
 a `Clicksend::Error` and isn't marked ambiguous, so neither the gem nor the recipe above can tell
 that the message may have gone out, and a job runner will retry the job and may send it twice. Use
-the client's own timeouts instead (for a job, e.g. `CLICKSEND.with(timeout: 10)`): a read timeout
-then raises an ambiguous `Clicksend::TimeoutError` that the recipe handles.
+the client's own timeouts instead: a read timeout then raises an ambiguous `Clicksend::TimeoutError`
+that the recipe handles.
+
+### Marking a send in flight
+
+A killed process (SIGKILL, out of memory, a lost host), a recovered job or a double enqueue re-runs
+a job however the timeouts are set, and the recipe can't see the first run. If a duplicate matters,
+record that the send is in flight **before** calling `deliver`, on the row that already represents
+the message in your database, and let a run that finds that mark reconcile instead of sending. The
+gem can't do this for you: only your database knows which jobs started. With a worker that died
+after ClickSend had accepted the message:
+
+| Approach | Messages sent |
+|---|---|
+| the recipe alone | 2 |
+| claim and send inside one database transaction | **2**: the dying worker's rollback erased the claim |
+| a conditional `UPDATE`, committed before `deliver` | 1; the re-run found the claim and sent nothing |
+| the same, with 4 runs of one job at once | 1 |
+
+The recipe with a marker:
+
+```ruby
+# notifications: sms_state (string, default "pending"), sms_claimed_at, sms_message_id
+class SendSmsJob < ApplicationJob
+  self.log_arguments = false
+  retry_on Clicksend::Error, attempts: 5, wait: :polynomially_longer # sees only failures that were not processed
+
+  def perform(notification_id)
+    # The claim commits on its own: never call this inside a transaction that also covers deliver.
+    claimed = Notification.where(id: notification_id, sms_state: "pending")
+      .update_all(sms_state: "sending", sms_claimed_at: Time.current) == 1
+    return unless claimed # done, failed, or another run is sending (or died sending): never send here
+
+    notification = Notification.find(notification_id)
+    message = CLICKSEND_JOBS.sms.deliver(to: notification.phone, body: notification.text, custom_string: "notification:#{notification_id}")
+    notification.update_columns(sms_state: "sent", sms_message_id: message.message_id)
+  rescue Clicksend::AmbiguousRequestError
+    Notification.where(id: notification_id).update_all(sms_state: "unknown") # reconciled later, never resent here
+  rescue Clicksend::MessageRejected
+    Notification.where(id: notification_id).update_all(sms_state: "failed")
+  rescue Clicksend::Error
+    Notification.where(id: notification_id, sms_state: "sending").update_all(sms_state: "pending") # not processed
+    raise # release the claim, and let retry_on try again
+  end
+end
+```
+
+Anything that fails after the claim, including the updates in the `rescue` clauses, leaves the row
+`sending`, and a re-run then sends nothing: the safe direction. A 429 that outlasted the gem's
+retries released the claim, and the job's retry sent the message once.
+
+### Reconciling with history
+
+With the marker, reconciliation needs no hand-over: rows left `sending` or `unknown` are the
+work list. Check them with [`search_history`](#when-a-sends-outcome-is-unknown) from a recurring
+job, not straight after the failure: ClickSend doesn't say how soon a sent message appears in
+history.
+
+```ruby
+# Run every few minutes (e.g. a Solid Queue recurring task or a cron job).
+class SmsReconciliationJob < ApplicationJob
+  def perform
+    Notification.where(sms_state: %w[sending unknown]).where("sms_claimed_at < ?", 10.minutes.ago).find_each do |notification|
+      found = CLICKSEND.sms.search_history(to: notification.phone, custom_string: "notification:#{notification.id}",
+        sent_after: notification.sms_claimed_at)
+      if found.any?
+        notification.update_columns(sms_state: "sent", sms_message_id: found.first.message_id)
+      elsif notification.sms_claimed_at < 1.day.ago
+        notification.update_columns(sms_state: "unresolved") # for a person to decide
+      end
+    end
+  end
+end
+```
+
+Not finding the message is **not** a reason to resend it: an empty result doesn't prove that
+nothing was sent. Keep checking for a while, then let a person or the user decide (for a login
+code, the user asking for a new one is usually safer). Each check reads at least one page of
+history, against ClickSend's undocumented rate limits.
+
+### Sidekiq without ActiveJob
+
+The recipe works the same way in a `Sidekiq::Job`. Sidekiq can also stop an ambiguous error that
+escapes `perform` from being retried:
+
+```ruby
+class SendSmsWorker
+  include Sidekiq::Job
 
-**Job runners are at-least-once.** If a worker is killed or shut down mid-send (Sidekiq's default
-shutdown timeout, 25s, is shorter than the gem's 30s read timeout), the job runs again and the
-gem never sees the first attempt's outcome. If a duplicate matters, record that a send is in
-flight before calling `deliver`, and reconcile instead of sending when a job finds that record
-already there. The gem can't do this for you: only your database knows which jobs started.
+  # Safety net: an ambiguous error goes to the Dead set instead of being retried.
+  sidekiq_retry_in { |_count, error, _job| :kill if error.is_a?(Clicksend::AmbiguousRequestError) }
+  sidekiq_retries_exhausted { |job, error| Sidekiq.logger.warn("#{job["class"]} #{job["jid"]} gave up: #{error.class}") }
+
+  def perform(notification_id)
+    # as SendSmsJob#perform above: rescue Clicksend::AmbiguousRequestError and reconcile
+  end
+end
+```
+
+In Sidekiq 8.1.7 (measured with a real Sidekiq process, and read in its source):
+- `:kill` puts the job in the Dead set and calls `sidekiq_retries_exhausted` and the death
+  handlers. **Pressing "Retry Now" on that dead job runs the send again**: reconcile first.
+- `:discard` drops the job and calls only the death handlers, so nothing is left in the Web UI to
+  investigate.
+- `nil` keeps the normal retry schedule.
+
+Prefer rescuing inside `perform` and keep `:kill` as the safety net. `sidekiq_retries_exhausted`
+also runs when the ordinary retries run out.
 
 ## Calling other ClickSend endpoints
 
@@ -543,7 +777,7 @@ response.data           # the envelope's "data" (frozen Hash/Array)
 response.response_code  # => "SUCCESS"
 response.http_status; response.headers; response.body
 
-client.request(:put, "/v3/sms/#{message_id}/cancel")
+client.request(:get, "/v3/statistics/sms", operation: "statistics.sms")
 client.request(:get, "/v3/sms/templates", query: {page: 2})
 
 # Any paginated list, as raw Hashes:
@@ -580,7 +814,6 @@ ActiveSupport::Notifications.subscribe("request.clicksend") do |event|
   event.payload
   # => {http_method: :post, path: "/v3/sms/send", operation: "sms.deliver", idempotent: false,
   #     attempts: 1, http_status: 200, response_code: "SUCCESS", ambiguous: false}
-  StatsD.distribution("clicksend.request", event.duration, tags: ["operation:#{event.payload[:operation]}"])
 end
 ```
 
@@ -594,20 +827,87 @@ contain no phone numbers or message text. Paths and `operation:` labels you pass
 are reported as you wrote them, minus any query string or fragment. (An exception object attached
 by ActiveSupport carries the response body of an API error, as `#body` does.)
 
-**OpenTelemetry and other generic HTTP instrumentation.** Auto-instrumentation of Faraday or
-Net::HTTP (for example `opentelemetry-instrumentation-faraday`) works below this gem and records the
-full request URL, query string included. `sms.history(to: ...)` sends `q=to:+61...`, so recipients'
-phone numbers can end up in your traces. Configure that instrumentation to drop or sanitise URLs and
-query strings, or exclude ClickSend's host from it. ClickSend-specific tracing built on this gem's
-`request.clicksend` events doesn't have the problem, because those payloads never contain a query
-string. A dedicated adapter that does this (`clicksend-opentelemetry`) is planned but not released.
+**Structured logs.** For `key=value` or JSON logs, subscribe to both events and keep the fields you
+want:
+
+```ruby
+ActiveSupport::Notifications.subscribe(/\.clicksend\z/) do |event|
+  p = event.payload
+  fields = {event: event.name, operation: p[:operation], method: p[:http_method], path: p[:path],
+            status: p[:http_status], code: p[:response_code], attempts: p[:attempts], ambiguous: p[:ambiguous],
+            attempt: p[:attempt], delay: p[:delay], error: p[:error_class] || p.dig(:exception, 0),
+            duration_ms: event.duration.round(1)}.compact
+  Rails.logger.info(fields.map { |key, value| "#{key}=#{value}" }.join(" "))
+end
+# event=request.clicksend operation=sms.deliver method=post path=/v3/sms/send status=200 code=SUCCESS attempts=1 ambiguous=false duration_ms=184.2
+# event=request.clicksend operation=sms.deliver method=post path=/v3/sms/send attempts=1 ambiguous=true error=Clicksend::TimeoutError duration_ms=15001.7
+```
+
+**Metrics.** Label by `operation` and an outcome, **never by `path`**: paths such as
+`/v3/sms/receipts/{message_id}` contain IDs, so every message would create new series.
+
+```ruby
+ActiveSupport::Notifications.subscribe("request.clicksend") do |event|
+  p = event.payload
+  outcome = if p[:ambiguous] then "ambiguous" # may have been processed: reconcile
+  elsif p[:exception] then "error"
+  else "ok" # MessageRejected is raised after this event, where you call deliver
+  end
+  tags = {operation: p[:operation] || "other", outcome: outcome}
+
+  StatsD.distribution("clicksend.request.duration", event.duration, tags: tags)  # statsd-instrument, milliseconds
+  # prometheus-client: REQUEST_SECONDS.observe(event.duration / 1000.0, labels: tags)
+  # Yabeda:            Yabeda.clicksend.request_duration.measure(tags, event.duration / 1000.0)
+end
+```
+
+Alert on `outcome=ambiguous` (each one is a send to reconcile) and on `retry.clicksend` events whose
+`error_class` is `Clicksend::RateLimitError`.
+
+**Rails 8.1 structured events.** The events plug into `Rails.event` through a
+`StructuredEventSubscriber` (`retry` is a legal method name):
 
-A `Clicksend::Client` is frozen after construction and holds no mutable state. Share one client
-across threads, Puma workers and Sidekiq jobs. Loggers and instrumenters are called on the calling
-thread and must be thread-safe.
+```ruby
+# config/initializers/clicksend.rb
+class ClicksendEvents < ActiveSupport::StructuredEventSubscriber
+  def request(event)
+    emit_event("clicksend.request", event.payload.slice(:operation, :http_status, :response_code, :attempts, :ambiguous)
+      .merge(duration_ms: event.duration.round(2)))
+  end
+
+  def retry(event)
+    emit_event("clicksend.retry", event.payload.slice(:operation, :attempt, :delay, :error_class, :http_status))
+  end
+end
+ClicksendEvents.attach_to :clicksend
+```
 
-The default Net::HTTP adapter opens a connection per request, which costs a TCP and TLS handshake
-each time. For high volumes, the `faraday-net_http_persistent` gem reuses connections:
+**OpenTelemetry.** Generic HTTP instrumentation works below this gem and records the request URL
+with its query string, which this gem keeps out of its own logs, errors and events. For
+`sms.history(to:)` and `sms.search_history`, that is the recipient's phone number. With
+`opentelemetry-instrumentation-faraday` 0.33.0 the span (named just `GET`) recorded
+`url.full=https://rest.clicksend.com/v3/sms/history?order_by=date%3Aasc&q=to%3A%2B61411111111`;
+`opentelemetry-instrumentation-net_http` 0.29.1 records the same query as `url.query`. Both also
+send a `traceparent` header to ClickSend. With those versions:
+- the Net::HTTP instrumentation skips ClickSend with `untraced_hosts: ["rest.clicksend.com"]`;
+- the Faraday instrumentation has no such option. Don't enable it, or wrap ClickSend calls in
+  `OpenTelemetry::Common::Utilities.untraced { ... }`, which suppresses every span inside the block.
+
+For ClickSend spans without that problem, use
+[`clicksend-opentelemetry`](companions/clicksend-opentelemetry): one span per call, with retries
+as events and ambiguity as an attribute, and never a query string, body or phone number. It is
+available in this repository and released separately from this gem, on its own version line; it
+is not on RubyGems yet.
+
+**Thread safety.** A `Clicksend::Client` is frozen after construction and holds no mutable state.
+Share one client across threads, Puma workers and Sidekiq jobs. Loggers and instrumenters are
+called on the calling thread and must be thread-safe.
+
+**Persistent connections.** The default Net::HTTP adapter opens a connection per request, which
+costs a TCP and TLS handshake each time. The `faraday-net_http_persistent` gem reuses connections.
+In a local benchmark (a loopback TLS server adding a simulated 25 ms round trip; ClickSend's own
+latency wasn't measured), the median send took 31 ms instead of 84 ms, and 8 threads used one
+connection each instead of one per request.
 
 ```ruby
 threads = ENV.fetch("RAILS_MAX_THREADS", 5).to_i # every thread that shares this client
@@ -615,12 +915,14 @@ CLICKSEND = Clicksend::Client.new(adapter: [:net_http_persistent, {pool_size: th
 ```
 
 The pool must be **at least as large as the number of threads sharing the client** (Puma's
-threads, Sidekiq's concurrency). A thread that can't get a connection in time fails with a
-timeout, and the gem can't tell that nothing was sent, so a send fails as ambiguous. With this
-adapter, a refused connection or connect timeout is also reported as possibly sent. Neither case
-sends anything twice, but both can leave a message unsent that the default adapter would have
-retried. Build the client once (`with` creates a new pool), and note that this adapter isn't part
-of this gem's test suite.
+threads, Sidekiq's concurrency). A thread that can't get a connection in time fails with a timeout:
+in the same local benchmark, 31 of 400 sends from 8 threads sharing `pool_size: 2` timed out
+waiting. Since 1.2 the gem recognises that wait, a refused connection and a connect timeout under
+this adapter as "not sent", so they are retried like any other failure that never reached
+ClickSend, but each still uses up a retry and adds latency. TLS errors and connection resets stay
+"possibly sent", because the adapter can raise them after the request was written. Build the
+client once (`with` creates a new pool). A real-socket spec in this gem's suite checks that the
+adapter never repeats a send by itself.
 
 ## Testing your application
 
@@ -655,6 +957,50 @@ Clicksend::Client.new(username: "test", api_key: "test", transport: fake,
   retry_policy: Clicksend::RetryPolicy.new(base_delay: 0, max_delay: 0))
 ```
 
+**Assert on what was sent.** Opt-in matchers and assertions read `fake.sent_messages` (accepted
+messages; rejected recipients don't count). The gem depends on neither framework: these files load
+only when your test suite requires them. Each key is optional and names a `SentMessage` attribute
+(`to`, `body`, `custom_string`, `from`, `list_id`, `scheduled_at`, `country`, `message_id`,
+`sent_at`). Values are matched with `===`, so strings, regexps and procs all work. **Without a
+count, exactly one message must match**, because a second one is the duplicate you are testing
+for. A failure lists what was sent, one line per message, at most ten messages.
+
+```ruby
+require "clicksend/testing/rspec" # in spec/spec_helper.rb; adds the matchers to every example group
+
+RSpec.describe "sign-in codes" do
+  let(:fake) { Clicksend::Testing::FakeAPI.new } # a new fake per example, so there is nothing to reset
+
+  it "texts the code once" do
+    fake.client.sms.deliver(to: "+61411111111", body: "Your code is 481516", custom_string: "otp:42") # your code, given fake.client
+
+    expect(fake).to have_sent_sms(to: "+61411111111", body: /481516/, custom_string: "otp:42")
+    expect(fake).to have_sent_sms(custom_string: "otp:42").once # also .twice, .times(n), .exactly(n).times
+    expect(fake).not_to have_sent_sms(to: "+61422222222")       # none matching, however many
+  end
+
+  it "texts nobody until asked" do
+    expect(fake).to have_sent_no_sms
+  end
+end
+```
+
+```ruby
+require "clicksend/testing/minitest"
+
+class SignInCodeTest < Minitest::Test # or ActiveSupport::TestCase
+  include Clicksend::Testing::MinitestAssertions
+
+  def test_texts_the_code_once
+    fake = Clicksend::Testing::FakeAPI.new
+    fake.client.sms.deliver(to: "+61411111111", body: "Your code is 481516", custom_string: "otp:42") # your code, given fake.client
+
+    assert_sms_sent fake, to: "+61411111111", body: /481516/, custom_string: "otp:42" # count: 2 for two
+    assert_no_sms_sent fake, to: "+61422222222"
+  end
+end
+```
+
 **Simulate failures, including the ambiguous ones.** For outcomes where it matters you must say
 whether ClickSend processed the request before the failure, which is exactly the question your
 code has to cope with:
@@ -668,15 +1014,47 @@ fake.fail_next(:connection_refused)         # never sent: the gem retries it tra
 fake.fail_next(status: 429, retry_after: 0) # rate limited: retried
 fake.fail_next(status: 401)
 fake.fail_next(:timeout, processed: true, path: "/v3/sms/send", times: 2) # only matching requests
+fake.fail_next(:interrupted, processed: false) # not ClickSend: your job runner stopping the worker (below)
+```
+
+**Simulate the worker being stopped mid-send.** Job runners re-run a job that was stopped while
+running. Sidekiq's shutdown pushes busy jobs back to the queue and raises into their threads, and a
+deploy's SIGTERM or a killed process does much the same. If that happens during a send, the re-run
+sends again unless your job recorded "sending" before the call. `fail_next(:interrupted, ...)` lets
+you test that. It models your job runner, not anything ClickSend does: the fake raises
+`Clicksend::Testing::SimulatedInterrupt`, either after ClickSend accepted the message
+(`processed: true`) or before it saw it (`processed: false`). It is an `Exception` but not a
+`StandardError`, so the client lets it through untouched and never retries it, and a
+`rescue => e` doesn't catch it. It is not an `Interrupt` either, so an unrescued one fails just
+that test.
+
+```ruby
+fake.fail_next(:interrupted, processed: true, path: "/v3/sms/send") # stopped after ClickSend accepted it
+expect { SendOtpJob.perform_now(otp.id) }.to raise_error(Clicksend::Testing::SimulatedInterrupt) # the runner requeues it
+SendOtpJob.perform_now(otp.id)                     # the re-run finds your "sending" marker
+expect(fake).to have_sent_sms(custom_string: "otp:#{otp.id}") # exactly one, not two
 ```
 
 The fake also serves receipts, replies (marked read as ClickSend documents; whether the cutoff is
 inclusive is the fake's guess) and the account. Exceptions raised by your stub blocks surface as
 `Clicksend::Testing::StubError`, never as a simulated ClickSend failure.
 
-It deliberately **doesn't serve history**: ClickSend doesn't say how soon a sent message appears
-there, and an always-current fake history would let a "not in history, so resend" rule pass its
-tests and send twice in production. Stub `GET /v3/sms/history` with the rows each test needs.
+It deliberately **doesn't serve history by itself**: ClickSend doesn't say how soon a sent message
+appears there, and an always-current fake history would let a "not in history, so resend" rule
+pass its tests and send twice in production. Say what history shows at each point of your test:
+
+```ruby
+fake.stub_history                                              # nothing (yet)
+fake.stub_history(fake.sent_messages.last)                     # this message, status "Sent"
+fake.stub_history(fake.sent_messages.last, status: "Cancelled")
+```
+
+Every history request then gets exactly those rows, whatever its filters.
+
+`client.sms.cancel` works on a message the fake accepted with a schedule still in the future, and
+the message then appears in `fake.cancelled_messages` (it stays in `sent_messages`). ClickSend
+doesn't document what it answers for any other message, so the fake raises `StubError` instead of
+guessing; `fake.stub(:put, "/v3/sms/#{id}/cancel") { ... }` states the answer your test assumes.
 
 The fake simplifies, so don't let your tests depend on these:
 - Recipients that aren't 6 to 15 digits (optionally after `+`) get `INVALID_RECIPIENT`; ClickSend's
@@ -711,7 +1089,8 @@ per-message status `COUNTRY_NOT_ENABLED`, which `deliver` raises as `MessageReje
 | Send SMS (single, batch, lists, scheduled) | `POST /v3/sms/send` | `sms.deliver`, `sms.deliver_batch` |
 | Delivery receipts | `GET /v3/sms/receipts[/{id}]`, `PUT /v3/sms/receipts-read` | `sms.receipts`, `sms.receipt`, `sms.mark_receipts_read` |
 | Replies (inbound SMS) | `GET /v3/sms/inbound`, `PUT /v3/sms/inbound-read[/{id}]` | `sms.inbound`, `sms.mark_inbound_read`, `sms.mark_inbound_message_read` |
-| Message history | `GET /v3/sms/history` | `sms.history` |
+| Cancel a scheduled SMS | `PUT /v3/sms/{message_id}/cancel` | `sms.cancel` |
+| Message history | `GET /v3/sms/history` | `sms.history`, `sms.search_history` |
 | Pushed receipts and replies (webhooks) | automation rules with the URL action | `Clicksend::Webhook` |
 | Account balance | `GET /v3/account` | `account.fetch` |
 | Testing without the network | | `Clicksend::Testing::FakeAPI` |
@@ -721,6 +1100,10 @@ per-message status `COUNTRY_NOT_ENABLED`, which `deliver` raises as `MessageReje
 Deliberately **not** wrapped:
 - Fax and post. ClickSend no longer offers them to new customers.
 - Email, payments and recharge, and reseller features.
+- Price quotes (`POST /v3/sms/price`). ClickSend says a quote sends no message, but not that it
+  is free of other effects (it returns a `message_id`); call it through `client.request`, which
+  treats it as not safe to repeat.
+- Cancelling every scheduled message (`PUT /v3/sms/cancel-all`).
 
 For broad coverage, use ClickSend's official SDK. Notes on ambiguities in ClickSend's
 documentation that affect this gem are in [docs/clicksend-api-notes.md](docs/clicksend-api-notes.md).