@alteriom/painlessmesh 2.0.3 → 2.1.0

package/CHANGELOG.md

@@ -7,6 +7,129 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [2.1.0] - 2026-09-15
+
+`sendToInternet()` you can build a notifier on (#463). One user's WhatsApp
+alerts through CallMeBot kept failing, and the hardware rig showed why in the
+library: a request whose reply was slow was sent again -- one call reached the
+server four times --, a long reply was cut before the service said what it did,
+and a chunked reply reached the application with its transfer framing. 2.1.0
+issues each request once, retries only what cannot arrive twice, honours
+Retry-After, gives every attempt one request id, and hands the application the
+whole result: status, the service's own words, whether a resend is safe, and
+how many attempts it took. **Upgrade if a node of yours sends to the
+Internet.**
+
+**One behaviour change to read before upgrading:** the library no longer
+decides from a reply's words whether a service did what you asked. A CallMeBot
+"Too many requests" page under HTTP 201 is `success == true`, as HTTP says;
+read the reply in your sketch with the new result callback, as
+`examples/sendToInternet/callmebot.h` does. Wire-compatible with 2.0 nodes: the
+new ack and request fields are optional and ignored by older nodes. Validated
+on the six-family hardware rig, including a real WhatsApp delivered through
+the mesh and a real CallMeBot refusal reaching the application intact.
+
+### Changed
+
+- **`sendToInternet()` no longer retries a request the server may already
+  have.** A read timeout or a connection lost after the request was sent used
+  to count as a "network error" and was retried up to three times; on the
+  hardware rig one timed-out send reached the server four times, which for a
+  message service is four messages, or one message and three refusals. The
+  gateway now tells the origin node whether the identical request may be sent
+  again (`GatewayAckPackage::retryable`, JSON `"retry"`), and says yes only
+  when it cannot have arrived -- connection refused, a failure before the
+  request went out -- or when the server said it did not take it: HTTP 429 and
+  503. A 500, 502 or 504 is no longer retried. The error of a transport
+  failure that was not retried ends "(the request may have reached the
+  server; not retried)", and resending is the application's decision.
+  Acks from gateways that predate the field keep their old reading, except
+  that a 2xx is never retried.
+- **The library no longer judges a response body.** 2.0.3 matched
+  CallMeBot's "Too many requests" page inside the gateway and reported that
+  HTTP 201 as a failure. Whether a reply means what an application wanted is
+  that service's language, not HTTP's, so the gateway now applies HTTP's
+  meaning of the status (200, 201, 202 and 204 succeed; 203, 205-299 are
+  unverified failures) and carries the body to the application instead. **A
+  CallMeBot rate-limit refusal under HTTP 201 is therefore `success == true`
+  again at the library level.** The sendToInternet example reads CallMeBot's
+  reply itself (`examples/sendToInternet/callmebot.h`); a sketch that relied
+  on 2.0.3's check should do the same with the new result callback.
+
+### Added
+
+- **`sendToInternet()` with an `InternetResult` callback.** Besides
+  `success`, `httpStatus` and `error`, the result carries `response` -- the
+  start of the response body as one line, on success as well as failure --
+  `retryable`, `attempts` and `messageId`. The three-argument callback is
+  unchanged. The response travels in the ack as `"resp"` and the error
+  excerpt keeps the end of a long body as well as its start, so a verdict
+  after an echo of the request survives (#463): the gateway keeps the first
+  512 and the last 256 bytes of a body, not only its start, and removes
+  chunked transfer framing -- CallMeBot answers chunked, and its reply reached
+  the application as "a6 ... Message queued ... 0". `PAINLESSMESH_HAS_INTERNET_RESULT`
+  is defined, so code that also builds against older releases can `#ifdef` it.
+- **Retry-After.** A 429 or 503 retry waits at least as long as the server's
+  `Retry-After` (delay-seconds); a server that asks for more than 60 s gets no
+  automatic retry, and the error says when it wants the request back. The
+  request's timeout moves with the wait, so the invited retry is not timed
+  out first.
+- **Request ids.** Every attempt at one call carries the same `X-Request-Id`
+  and `Idempotency-Key` header (`pm-<origin>-<messageId>-<nonce>`), so a
+  service that honours idempotency keys drops a copy and a log can count
+  retries. The nonce is drawn per call and travels in the request (`"nonce"`),
+  so ids stay unique after the 16-bit message-id counter wraps; a request from
+  an older node keeps `pm-<origin>-<messageId>`.
+  Message ids now start at a random point each boot, so the first request
+  after a reboot does not reuse the key of the boot before.
+- **Test point:** the ledger counts requests per tag and records their
+  request ids, and `/retry-after/{seconds}` refuses a tag once with 429 and
+  `Retry-After`, then accepts it and records whether the retry came early.
+- **sendToInternet example:** alerts once per O2 episode and at most every
+  10 minutes, backs off 15 minutes after a CallMeBot refusal, tags the
+  startup message per boot, URL-encodes the phone number, and no longer
+  prints the API key. `callmebot.h` names CallMeBot's "APIKey is invalid"
+  reply (seen from the hardware rig) instead of calling it unrecognised.
+
+### Fixed
+
+- **A gateway's own request completed inside its HTTP handler.** A bridge or
+  shared gateway serving its own `sendToInternet()` delivered the result from
+  inside the gateway handler, while its `HTTPClient`, `WiFiClient` and response
+  buffers were still allocated; on an ESP8266 with ~11 KB free the
+  application's callback could not allocate what it built (hardware rig). The
+  local acknowledgment now completes from the scheduler, after the handler has
+  freed.
+- **`InternetResult::attempts` and `retryable` describe what was sent.**
+  `attempts` counts only a request that left the node, and a request that
+  never did -- refused for want of a mesh or a gateway, or whose every routing
+  attempt failed -- is reported safe to resend.
+
+- **`SentBuffer::requestLength(0)` answered 1, not 0.** `buffer_length - 1`
+  leaves room for the terminator `toCharArray()` always writes; on an
+  unsigned zero it wraps to `SIZE_MAX`, `min()` then picks the message, and
+  a caller with no room at all is told one byte is available -- which the
+  method's own contract forbids ("<= the requested length") and `read()`
+  acts on, writing `length + 1` bytes. Found because the randomised buffer
+  scenario draws its length from `runif(0, ...)`: it asked for zero about
+  once in sixty runs and failed CI on pull requests nowhere near the buffer.
+
+- **A node that lost its last connection waited 15 s before it scanned**
+  (#459). `connectToAP()` logs "scan rate set to fast" and sets a
+  `0.5 * SCAN_INTERVAL` period -- fifteen seconds, and the shortest of a set
+  of long ones; "slow" is four intervals. For a node that still has a live
+  connection that is soon enough. For one with none it is the whole cost of
+  the outage, and it is paid while its neighbours still route to it until
+  their own `NODE_TIMEOUT`, so unicasts addressed to it are accepted by the
+  sender and dropped. Caught on the hardware rig: an ESP32-C3 closed its only
+  uplink on a momentary nodeSync contradiction, logged the "fast" line 12 ms
+  later, and did not scan for exactly 15 000 ms -- 16.2 s out of the mesh,
+  answering an empty node list, while the scan that eventually ran found five
+  mesh APs at -30 to -58 dBm and associated 1.2 s later. A node with no live
+  connections now scans at once, once per outage (`layout::liveSubs()` is the
+  test, and a node that is simply alone falls back to the interval rather
+  than scanning back to back).
+
 ## [2.0.3] - 2026-09-11
 
 `sendToInternet()` fixes from one user's WhatsApp integration (#450, #452,

package/CONTRIBUTING.md

@@ -2,13 +2,55 @@
 
 ## Branches
 
-- `main` holds released code. A push to `main` that carries a version bump,
-  or whose head commit message starts with `release:`, is what tags and
-  publishes a release (see [RELEASE_GUIDE.md](RELEASE_GUIDE.md)).
-- `Feat/next-release` is the integration branch for the next version. Open
-  pull requests against it.
-- Work happens on short-lived feature branches (`fix/…`, `feat/…`, `docs/…`)
-  cut from `Feat/next-release`.
+| Branch | What it is | Lifetime |
+|---|---|---|
+| `main` | the released line. A push carrying a version-file change, or whose head commit starts with `release:`, tags and publishes (see [RELEASE_GUIDE.md](RELEASE_GUIDE.md)) | permanent |
+| `release/<major>.x` | one major line — `release/3.x` while 3.0 is being built, and the same branch afterwards for patches to 2.x once `main` has moved on | as long as that major is being built or supported |
+| `release/<version>` | preparing one release: the version files, the changelog date | short-lived |
+| `fix/…` `feat/…` `docs/…` `ci/…` `test/…` `refactor/…` `chore/…` | one change | short-lived |
+
+**Open pull requests against `main`** unless the change belongs to a major
+line that has its own branch, in which case open them against that. This is
+also what the repository does in practice: every fix since #448 was merged to
+`main`.
+
+`Feat/next-release` is retired. It was where v2 was built while `main` still
+carried v1, which is exactly the job `release/<major>.x` now names; it holds
+nothing `main` does not, and leaving it there sent contributors at a branch
+27 commits behind.
+
+### Naming
+
+- **`type/short-slug`**, lowercase, hyphens — never underscores or capitals.
+  The type is one of the [Conventional Commits](https://www.conventionalcommits.org/)
+  types this repository already uses in commit subjects, so a branch and the
+  commits on it agree about what they are.
+- **An issue number goes at the end** when there is one: `fix/rejoin-459`.
+  A slug that says only the area (`fix/critical-bugs`) tells a later reader
+  nothing about what was wrong.
+- **No `v` in a branch name.** Tags carry it (`v2.0.3`); branches do not
+  (`release/2.0.4`, `release/3.x`). 2.0.2 shipped from `release/2.0.2` and
+  2.0.3 from `release/v2.0.3` — pick the one without.
+- **Tool-generated prefixes** (`copilot/…`, `claude/…`, `dependabot/…`) are
+  left as the tool makes them. They are ordinary short-lived work branches
+  and are deleted on merge like any other.
+
+### Status
+
+A branch is deleted when its pull request merges — by whoever merges it, or
+by GitHub's automatic branch deletion. Anything still on the remote is
+either live work or something that was forgotten, and the two look identical
+from a branch list, so:
+
+```bash
+./scripts/branch-status.sh          # every remote branch: ahead, behind, age, unmerged work
+./scripts/branch-status.sh --stale  # only the ones with nothing of their own
+```
+
+`ahead` counts commits `main` does not have; `unmerged` counts those whose
+change is not in `main` under any commit, which is the number that matters
+after a squash merge. A branch with `unmerged=0` can be deleted without
+losing anything.
 
 Maintainers merge quickly, often as a squash. Push every commit you describe
 before you describe it, and cut follow-up work from the merged base rather
@@ -16,7 +58,8 @@ than from a stale branch.
 
 ## Submit a pull request
 
-- Point the pull request at `Feat/next-release`, not `main`.
+- Point the pull request at `main`, or at the major line's own branch when
+  the change belongs to one (see [Branches](#branches)).
 - Say what was wrong, how you know (a log, a test, a measurement), and what
   the change does about it. For anything that touches the radio, routing,
   the gateway or OTA, the evidence is a serial log or a run on the

package/examples/sendToInternet/README.md

@@ -64,6 +64,55 @@ if (mesh.hasInternetConnection()) {
 }
 ```
 
+### Reading the reply: the whole result
+
+`success` means HTTP 200, 201, 202 or 204 -- what HTTP calls a success. It
+does not mean the service did what you wanted: CallMeBot, for one, answers
+"Too many requests" under HTTP 201. Pass a callback that takes an
+`InternetResult` to get the start of the response body as well, and decide
+in your sketch:
+
+```cpp
+#include "callmebot.h"  // from this example
+
+mesh.sendToInternet(url, "", [](const painlessmesh::InternetResult& result) {
+  // result.messageId, result.success, result.httpStatus, result.error,
+  // result.response (the body, as one line), result.retryable, result.attempts
+  const auto reply = callmebot::judge(result.httpStatus, result.response);
+  if (reply.accepted) {
+    Serial.println("CallMeBot queued the message");
+  } else {
+    Serial.printf("Not sent: %s -- CallMeBot said: %s\n", reply.meaning,
+                  result.response.c_str());
+  }
+});
+```
+
+`callmebot.h` is part of the example, not the library: it knows CallMeBot's
+wording ("Message queued", "Too many requests", "Your Account is Paused",
+"APIKey is invalid")
+and that its HTTP 208 has not meant a delivery. The desktop test suite checks
+it against the CallMeBot-shaped test point in `test/mock-http-server/`.
+
+### Retries and duplicates
+
+The library issues each request **once**, and retries only when a retry
+cannot deliver it twice:
+
+| What happened | Retried? |
+|---|---|
+| Connection refused, or a send failure before the request went out | yes |
+| Gateway without Internet, a host that does not resolve | no -- a retry within seconds fails the same way |
+| HTTP 429 or 503 (the server did not take it) | yes, not before `Retry-After` |
+| Read timeout, connection lost after sending | **no** -- the server may have it |
+| Any other status, including every 2xx and 500 | no |
+
+A request that was not retried reports `retryable == false`; a transport error
+says "(the request may have reached the server; not retried)". Resending is
+then your call, knowing it may arrive twice. Every attempt at one call carries
+the same `X-Request-Id` and `Idempotency-Key` header, so a service that honours
+idempotency keys drops the copy, and a log can tell a retry from a second send.
+
 ### Sending JSON to REST API
 
 ```cpp
@@ -143,7 +192,8 @@ mesh.sendToInternet("https://api.callmebot.com/...", "", callback);
 
 1. Verify your Callmebot API key is correct
 2. Ensure phone number includes country code (e.g., `+1234567890`)
-3. Check HTTP status code in callback (200 = success)
+3. Print `result.response` in the callback: CallMeBot says why in the body,
+   and HTTP 200/201 alone does not mean it sent anything
 4. URL-encode special characters in the message
 
 ### Gateway shows "no internet access" but WiFi is connected
@@ -188,7 +238,9 @@ The callback provides `httpStatus` to indicate the result:
 **FAILURE (success = false):**
 - `203 Non-Authoritative Information` - **Cached/proxied response, NOT actual delivery**
 
-⚠️ **The status alone does not say whether WhatsApp got the message.** CallMeBot answers a refusal (for example "Too many requests") with HTTP 201 or HTTP 203, the same error page under both. The gateway therefore reads the start of the response body: a 2xx whose body says the service refused the request is reported as a failure, with the service's own words in the callback's `error` string, and HTTP 203 stays a failure because a proxy, not the WhatsApp API, may have answered. If your callback prints `HTTP 201: service refused the request: Oops! Too many requests...`, that is CallMeBot talking, and the message was **NOT delivered**. Likewise `HTTP 208: not a delivery the gateway can confirm: ...`: CallMeBot answers 208 to messages that never arrive (issues #450 and #452), so only 200, 201, 202 and 204 with a clean body are reported as sent, and the rest of the line is whatever the service said.
+⚠️ **The status alone does not say whether WhatsApp got the message.** CallMeBot answers a refusal (for example "Too many requests") with HTTP 201 or HTTP 203, the same error page under both, and it has answered HTTP 208 to messages that never arrived (issues #450 and #452). The library reports HTTP's meaning -- 201 is a success, 203 and 208 are not -- and hands your callback the start of the body in `result.response`. The sketch's `callmebot::judge()` reads that body: `✅ WhatsApp message queued by CallMeBot` only when CallMeBot said "Message queued", and otherwise `❌ WhatsApp message not sent` followed by what CallMeBot said.
+
+**The sketch paces its alerts.** CallMeBot is free and rate-limited. The sketch sends one WhatsApp when O2 falls below the threshold, not one per reading, never more than one every 10 minutes per node, and waits 15 minutes after "Too many requests" or a paused account. Several nodes with the same API key share CallMeBot's limit; raise `ALERT_MIN_INTERVAL_MS` accordingly.
 
 Also replace `CLOUD_URL` at the top of the sketch: the placeholder `api.example.com` does not resolve, and until you change it the sketch skips the cloud send and says so instead of reporting `connection refused` every minute.
 

package/examples/sendToInternet/callmebot.h

@@ -0,0 +1,139 @@
+//************************************************************
+// callmebot.h - what a CallMeBot WhatsApp reply means
+//
+// painlessMesh reports what HTTP says: the status, whether a retry could
+// deliver the request twice, and the start of the response body. Whether
+// WhatsApp will actually show the message is CallMeBot's business, and
+// CallMeBot does not put it in the status code:
+//
+//   * a rate-limit refusal comes back as HTTP 201 or HTTP 203 with an
+//     "Oops! Too many requests" page (probed 2026-09-10, painlessMesh #450);
+//   * HTTP 208 came back four times for messages that never arrived (#452);
+//   * a paused account answers with the request echoed back and, at the end,
+//     "Your Account is Paused ... send the word 'resume'" (#463);
+//   * an API key CallMeBot does not know answers HTTP 203 with the request
+//     echoed back and "APIKey is invalid. Please create a new one ..."
+//     (observed from the hardware rig, 2026-09-15);
+//   * a message CallMeBot accepted answers "Message queued".
+//
+// So the sketch reads the reply with judge() below. It is plain C++ over the
+// status and the body text, with no Arduino or network calls, so the desktop
+// test suite checks it (test/catch/catch_callmebot_example.cpp) against the
+// same CallMeBot-shaped test point the hardware farm uses.
+//
+// The phrases are CallMeBot's, observed or documented at the dates above. If
+// CallMeBot changes its wording, a reply lands in Verdict::Unrecognized and
+// the sketch prints what the service said -- it never guesses "sent".
+//************************************************************
+#ifndef SENDTOINTERNET_CALLMEBOT_H
+#define SENDTOINTERNET_CALLMEBOT_H
+
+#include "painlessmesh/gateway.hpp"
+
+namespace callmebot {
+
+enum class Verdict {
+  /** CallMeBot accepted the message ("Message queued"). */
+  Queued,
+  /** "Too many requests": nothing was sent; wait before the next message. */
+  RateLimited,
+  /** The account is paused: send "resume" to the bot; resending won't help. */
+  AccountPaused,
+  /** The API key is not one CallMeBot knows: get a new one; resending won't help. */
+  InvalidApiKey,
+  /** HTTP 208: in the field this never meant a delivery, whatever the body. */
+  NotDelivered,
+  /** No HTTP reply at all; the library's error says why. */
+  NoReply,
+  /** A reply this file does not know. Treated as not sent. */
+  Unrecognized
+};
+
+struct Judgement {
+  Verdict verdict;
+  /** True only for Verdict::Queued. */
+  bool accepted;
+  /**
+   * Minimum wait before the sketch sends CallMeBot anything else, in ms.
+   * Non-zero after a refusal, so a sketch does not turn one refusal into a
+   * stream of them.
+   */
+  uint32_t holdOffMs;
+  /** One line for the serial log. */
+  const char* meaning;
+};
+
+/** How long to leave CallMeBot alone after it said "Too many requests". */
+static const uint32_t RATE_LIMIT_HOLD_OFF_MS = 15UL * 60UL * 1000UL;
+
+inline bool isHttpSuccess(uint16_t status) {
+  return status == 200 || status == 201 || status == 202 || status == 204;
+}
+
+/**
+ * Read a CallMeBot reply.
+ *
+ * @param httpStatus InternetResult::httpStatus (0 when no reply arrived)
+ * @param response   InternetResult::response, the start of the body
+ */
+inline Judgement judge(uint16_t httpStatus, const TSTRING& response) {
+  using painlessmesh::gateway::responseContains;
+
+  if (httpStatus == 0) {
+    return {Verdict::NoReply, false, 0,
+            "no reply from CallMeBot; it may or may not have received the request"};
+  }
+  // Refusals first: they arrive under success statuses too.
+  if (responseContains(response, "Too many requests")) {
+    return {Verdict::RateLimited, false, RATE_LIMIT_HOLD_OFF_MS,
+            "CallMeBot refused: too many requests"};
+  }
+  if (responseContains(response, "Account is Paused") ||
+      responseContains(response, "send the word 'resume'")) {
+    return {Verdict::AccountPaused, false, RATE_LIMIT_HOLD_OFF_MS,
+            "CallMeBot account paused: send 'resume' to the CallMeBot bot on WhatsApp"};
+  }
+  if (responseContains(response, "APIKey is invalid")) {
+    // Nothing a retry changes: hold off like a refusal, so a sketch with a
+    // wrong key does not call the API once a minute until someone notices.
+    return {Verdict::InvalidApiKey, false, RATE_LIMIT_HOLD_OFF_MS,
+            "CallMeBot rejected the API key: get a new one from CallMeBot and set WHATSAPP_APIKEY"};
+  }
+  // Before the success text: a 208 carrying "Message queued" still did not
+  // arrive in the field.
+  if (httpStatus == 208) {
+    return {Verdict::NotDelivered, false, 0,
+            "CallMeBot answered HTTP 208; such messages have not arrived"};
+  }
+  if (isHttpSuccess(httpStatus) && responseContains(response, "Message queued")) {
+    return {Verdict::Queued, true, 0, "CallMeBot queued the message"};
+  }
+  return {Verdict::Unrecognized, false, 0,
+          "CallMeBot's reply is not one this sketch recognises; not counted as sent"};
+}
+
+/**
+ * The URL with the API key replaced, for printing. A serial log gets pasted
+ * into issues; the key should not go with it.
+ */
+inline TSTRING redactApiKey(const TSTRING& url) {
+#if defined(PAINLESSMESH_BOOST)
+  const auto start = url.find("apikey=");
+  if (start == TSTRING::npos) return url;
+  const auto valueStart = start + 7;
+  auto end = url.find('&', valueStart);
+  if (end == TSTRING::npos) end = url.length();
+  return url.substr(0, valueStart) + "***" + url.substr(end);
+#else
+  const int start = url.indexOf("apikey=");
+  if (start < 0) return url;
+  const int valueStart = start + 7;
+  int end = url.indexOf('&', valueStart);
+  if (end < 0) end = url.length();
+  return url.substring(0, valueStart) + "***" + url.substring(end);
+#endif
+}
+
+}  // namespace callmebot
+
+#endif  // SENDTOINTERNET_CALLMEBOT_H