span-panel-api 3.0.0b8__tar.gz → 3.0.0b10__tar.gz

{span_panel_api-3.0.0b8 → span_panel_api-3.0.0b10}/CHANGELOG.md

@@ -4,6 +4,37 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [3.0.0b10]
+
+### Changed
+
+- **The wait for a panel to finish rebooting no longer gives up.** It used to stop after a fixed number of attempts, and that bound was wrong twice for the same reason: it was sized against a reboot somebody had measured, and the next reboot was not that
+  reboot. Giving up has nothing to recommend it — the only things that start another attempt are the reconnect edge and the panel republishing its data-model version, and a panel that finishes booting after the wait expired produces neither, so running out
+  of attempts means stranded until somebody reloads by hand. It now waits as long as the panel takes.
+- **Waiting costs nothing you were relying on.** Energy sensors already hold their last reading through an outage on their own grace period — fifteen minutes by default, configurable — which exists precisely so a gap does not become an `unknown` and a
+  statistics spike. That is untouched by how long this waits, and it was the only thing that would have justified a deadline. What is left is one request every thirty seconds to a device on your own network.
+- **The retry interval settles at thirty seconds rather than growing.** Backing off without a ceiling would mean a panel that took a while to return was then ignored for longer than it took. The gap goes 1, 2, 4, 8, 16, 30 and stays there, so once your
+  panel is answering it is noticed within half a minute however long the wait has already run.
+
+### Fixed
+
+- **Four more ways a booting panel answers now count as "not ready" rather than as a hard failure.** b8 and b9 covered the 502 that a live upgrade produced; review found the fix had covered the observed shape rather than the class. A panel resetting its
+  listener mid-request raises `ReadError` or `WriteError`, a proxy that dies mid-request raises `RemoteProtocolError`, and a panel part-way through starting can answer `200` with a truncated or empty body. All four escaped untranslated, skipped the retry
+  entirely, and stranded the parser exactly as the 502 did. Transport failures are now `SpanPanelConnectionError` and an unusable body is `SpanPanelServerError`.
+- **The last retry attempt happens after the reboot it is sized for.** The window ended with a sleep that no attempt followed: it read 241 seconds while the final request went out at 211, so a panel ready at 220 was still abandoned. The loop no longer
+  sleeps after its final attempt — which also stopped it holding the in-flight guard, and the warning, for a pointless extra backoff — and the last request now lands at 241 seconds. The test asserts that offset instead of summing the sleeps, which was
+  restating the implementation's own off-by-one.
+- **The give-up warning no longer promises a recovery that cannot arrive.** It said data would read as missing "until the next reconnect". The triggers are the reconnect edge and the retained `data-model-version` message, and a panel that finishes booting
+  produces neither again, so exhausting the window means stuck until a reload. It now says so.
+
+## [3.0.0b9]
+
+### Fixed
+
+- **The retry window for a rebooting panel is actually widened this time.** b8 taught the schema fetch to treat a `502` as "not ready yet" but shipped with the old five attempts capped at eight seconds — about twenty-three seconds against a panel observed
+  taking four minutes to come back, still answering 502 when the broker returned. The widening was written, lost to a failed edit in the same change, and shipped without it; nothing failed, because catching the 502 and giving up early looks exactly like
+  working. Now twelve attempts backing off to thirty seconds, a little over four minutes, and pinned by a test that asserts the total window outlasts the observed reboot rather than checking the constants individually.
+
 ## [3.0.0b8]
 
 ### Fixed

{span_panel_api-3.0.0b8 → span_panel_api-3.0.0b10}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: span-panel-api
-Version: 3.0.0b8
+Version: 3.0.0b10
 Summary: A client library for SPAN Panel API
 Project-URL: Homepage, https://github.com/SpanPanel/span-panel-api
 Project-URL: Issues, https://github.com/SpanPanel/span-panel-api/issues

{span_panel_api-3.0.0b8 → span_panel_api-3.0.0b10}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "span-panel-api"
-version = "3.0.0b8"
+version = "3.0.0b10"
 description = "A client library for SPAN Panel API"
 authors = [
     {name = "SpanPanel"}

{span_panel_api-3.0.0b8 → span_panel_api-3.0.0b10}/src/span_panel_api/auth.py

@@ -188,10 +188,18 @@ async def get_homie_schema(
     try:
         async with _get_client(httpx_client, timeout) as client:
             response = await client.get(url)
-    except httpx.ConnectError as exc:
-        raise SpanPanelConnectionError(f"Cannot reach panel at {host}") from exc
     except httpx.TimeoutException as exc:
         raise SpanPanelTimeoutError(f"Timed out connecting to {host}") from exc
+    except httpx.TransportError as exc:
+        # Every way the connection itself can fail, not just a refused connect:
+        # `ReadError` and `WriteError` when a rebooting panel resets mid-request,
+        # and `RemoteProtocolError` when its proxy closes without answering --
+        # which is exactly what a proxy restarting under load produces. Catching
+        # only `ConnectError` meant those escaped this function untranslated,
+        # skipped the caller's retry clause entirely, and stranded the parser the
+        # same way a 502 used to. `TimeoutException` is itself a `TransportError`,
+        # so it has to be caught first.
+        raise SpanPanelConnectionError(f"Cannot reach panel at {host}: {exc}") from exc
 
     if response.status_code >= 500:
         # A rebooting panel answers 502 from its front end while the application
@@ -210,7 +218,24 @@ async def get_homie_schema(
             status_code=response.status_code,
         )
 
-    data: dict[str, object] = response.json()
+    try:
+        parsed = response.json()
+    except ValueError as exc:
+        # A panel part-way through starting can answer 200 with a truncated or
+        # empty body. Retryable for the same reason a 502 is -- it is "not ready
+        # yet" wearing a different status -- and untranslated this had precisely
+        # the 502's old character: raised out of the caller's retry loop on the
+        # first attempt and left the parser where it was.
+        raise SpanPanelServerError(
+            f"Panel not ready: {host} answered 200 with a body that is not JSON",
+            status_code=response.status_code,
+        ) from exc
+    if not isinstance(parsed, dict):
+        raise SpanPanelServerError(
+            f"Panel not ready: {host} answered 200 with {type(parsed).__name__}, not an object",
+            status_code=response.status_code,
+        )
+    data: dict[str, object] = parsed
 
     # Extract types — each value is a dict of property definitions
     raw_types = data.get("types", {})

{span_panel_api-3.0.0b8 → span_panel_api-3.0.0b10}/src/span_panel_api/exceptions.py

@@ -30,7 +30,13 @@ class SpanPanelAPIError(SpanPanelError):
 
 
 class SpanPanelServerError(SpanPanelAPIError):
-    """Server error (500)."""
+    """The panel answered, and the answer means "not ready yet".
+
+    Any 5xx, and a 200 whose body cannot be a schema. Distinct from
+    `SpanPanelAPIError` because a caller can retry this and should not retry a
+    4xx, which will not fix itself. A rebooting panel produces these for as long
+    as its front end is up and the application behind it is not.
+    """
 
 
 class SpanPanelStaleDataError(SpanPanelError):

{span_panel_api-3.0.0b8 → span_panel_api-3.0.0b10}/src/span_panel_api/mqtt/client.py

@@ -48,9 +48,21 @@ _CIRCUIT_NAMES_POLL_INTERVAL_S = 0.25
 # Re-reading the schema after a suspected generation change. Bounded because the
 # caller is a fire-and-forget task on a live connection, and generous enough to
 # outlast a panel that is still binding its HTTP port after a restart.
-_REDISPATCH_RETRY_ATTEMPTS = 5
 _REDISPATCH_RETRY_INITIAL_S = 1.0
-_REDISPATCH_RETRY_MAX_S = 8.0
+_REDISPATCH_RETRY_MAX_S = 30.0
+_REDISPATCH_LOG_EVERY = 20
+"""How long to wait for the panel's HTTP endpoint after it returns on MQTT.
+
+Sized from a live firmware upgrade rather than guessed. The panel dropped MQTT at
+11:22:07 and the broker was back at 11:26:15 -- four minutes -- and its HTTP
+front end was still answering 502 at that moment. Five attempts capped at 8s
+gives up after about 23 seconds, which is not the same order of magnitude as a
+device that is still booting: catching the 502 buys nothing if the loop stops
+before the panel is ready.
+
+Twelve attempts backing off to 30s is a little over four minutes. Each one is a
+single GET, and the panel is the only thing that can end the wait.
+"""
 
 
 def _metadata_for_the_log() -> tuple[list[str], str]:
@@ -805,7 +817,7 @@ class SpanMqttClient:
         return active != observed
 
     async def _fetch_schema_with_retry(self) -> V2HomieSchema | None:
-        """Read the panel's REST schema, allowing for HTTP trailing the broker.
+        """Read the panel's REST schema, waiting for HTTP to catch up with the broker.
 
         A panel that has just restarted accepts MQTT before it serves HTTP — the
         broker is listening while the application is still binding its port. The
@@ -814,40 +826,56 @@ class SpanMqttClient:
         was no further edge to retry on, leaving the wrong parser in place for the
         rest of the session.
 
-        So this waits, briefly and boundedly. Returning None rather than raising
-        because the caller's job is to reconsider the parser, and being unable to
-        is not a reason to disturb a connection that is otherwise working.
+        **This waits as long as it takes, and that is deliberate.** Every bounded
+        version of it has been wrong, twice for the same reason: the bound was
+        sized against a reboot somebody had measured, and the next reboot was not
+        that reboot. Giving up has no upside to weigh against being wrong. The
+        triggers for another attempt are the reconnect edge and the retained
+        `data-model-version` message, and a panel that finishes booting after the
+        loop gave up produces neither — so exhausting a bound does not mean
+        "try again later", it means stranded until somebody reloads by hand.
+
+        Nor does waiting cost the freshness of anything. Energy sensors already
+        hold their last valid reading through an outage on their own grace period,
+        which exists precisely so a gap does not become an `unknown` and a
+        statistics spike; that mechanism is untouched by how long this waits, and
+        it is the thing that would have justified a deadline here. What is left is
+        one HTTP GET every thirty seconds to a device on the local network, which
+        is less traffic than the ordinary snapshot poll.
+
+        Ends on success, on cancellation — `close()` cancels this task, so unload
+        and shutdown are prompt — or on an error that is not the panel still
+        coming up, which is left to raise.
         """
         delay = _REDISPATCH_RETRY_INITIAL_S
-        last: Exception | None = None
-        for _ in range(_REDISPATCH_RETRY_ATTEMPTS):
+        attempts = 0
+        while True:
             try:
                 return await get_homie_schema(self._host, port=self._panel_http_port, httpx_client=self._httpx_client)
             except (
                 SpanPanelConnectionError,
                 SpanPanelTimeoutError,
-                # The third way HTTP lags the broker, and the one a real upgrade
-                # actually produced: the panel answers, with 502. Its front end is
-                # up while the application behind it is still starting, which is
-                # the ordinary order for a booting device. Omitting this meant the
-                # first attempt raised straight out of this loop, out of the
-                # fire-and-forget task that called it, and the parser was never
-                # swapped -- observed on two Home Assistant instances watching one
-                # panel through the same upgrade, neither of which recovered
-                # without a manual reload.
+                # The panel answering rather than refusing: a 5xx from its front
+                # end while the application behind it starts, or a 200 carrying a
+                # body that cannot be a schema. The ordinary shape of a reboot,
+                # because a device brings its network stack and proxy up before
+                # its application -- and the shape that stranded two live installs
+                # when it was not caught here.
                 SpanPanelServerError,
             ) as exc:
-                last = exc
+                attempts += 1
+                if attempts == 1 or attempts % _REDISPATCH_LOG_EVERY == 0:
+                    # First failure, then occasionally. A panel that never returns
+                    # would otherwise write a line every thirty seconds forever,
+                    # and the second line is worth no more than the first.
+                    _LOGGER.warning(
+                        "Panel is not serving its schema yet (%s). Attempt %d; still "
+                        "waiting, and the parser stays as it is until it answers.",
+                        exc,
+                        attempts,
+                    )
                 await asyncio.sleep(delay)
                 delay = min(delay * 2, _REDISPATCH_RETRY_MAX_S)
-        _LOGGER.warning(
-            "Could not re-read the panel schema after %d attempts (%s). The active "
-            "parser is unchanged; if the panel's schema generation did change, its "
-            "data will read as missing until the next reconnect.",
-            _REDISPATCH_RETRY_ATTEMPTS,
-            last,
-        )
-        return None
 
     async def _redispatch_if_generation_changed(self) -> None:
         """Swap the parser when the panel comes back as a different schema generation.

{span_panel_api-3.0.0b8 → span_panel_api-3.0.0b10}/tests/test_auth_and_homie_helpers.py

@@ -221,3 +221,55 @@ class TestHttpxClientInjectionAuthHelpers:
 
         mock_cls.assert_not_called()
         injected.aclose.assert_not_called()
+
+
+class TestGetHomieSchemaNotReadyShapes:
+    """Every way a booting panel answers that is not a clean 5xx.
+
+    Each of these used to escape `get_homie_schema` untranslated, skip the
+    caller's retry clause entirely, and strand the parser — the same failure the
+    502 produced on a live upgrade, wearing a different exception.
+    """
+
+    @pytest.mark.asyncio
+    @pytest.mark.parametrize(
+        "failure",
+        [
+            httpx.ReadError("connection reset"),
+            httpx.WriteError("broken pipe"),
+            httpx.RemoteProtocolError("server closed connection without sending a response"),
+        ],
+        ids=["read-reset", "write-reset", "proxy-closed-without-answering"],
+    )
+    async def test_a_transport_failure_is_a_connection_error(self, failure: Exception) -> None:
+        """A panel resetting its listener mid-request, and a proxy dying mid-request.
+
+        `httpx.TimeoutException` is itself a `TransportError`, so the timeout
+        branch has to stay ahead of this one — covered by the timeout test above.
+        """
+        with patch("span_panel_api._http.httpx.AsyncClient") as cls:
+            cls.return_value = _mock_client("get", failure)
+            with pytest.raises(SpanPanelConnectionError):
+                await get_homie_schema("192.168.1.1")
+
+    @pytest.mark.asyncio
+    @pytest.mark.parametrize("body", ["", "{trunc", "null", "[]"], ids=["empty", "truncated", "null", "list"])
+    async def test_a_200_that_cannot_be_a_schema_is_not_ready_rather_than_broken(self, body: str) -> None:
+        """A panel part-way through starting can answer 200 with nothing usable.
+
+        Retryable for the same reason a 502 is: it is "not ready yet" wearing a
+        success status. The bounded attempt count makes retrying a genuinely
+        broken body cheap.
+        """
+        response = MagicMock()
+        response.status_code = 200
+        response.json = MagicMock(side_effect=(lambda: json.loads(body)) if body else ValueError("no content"))
+        mock = AsyncMock()
+        mock.get = AsyncMock(return_value=response)
+        mock.__aenter__ = AsyncMock(return_value=mock)
+        mock.__aexit__ = AsyncMock(return_value=False)
+
+        with patch("span_panel_api._http.httpx.AsyncClient") as cls:
+            cls.return_value = mock
+            with pytest.raises(SpanPanelServerError):
+                await get_homie_schema("192.168.1.1")

{span_panel_api-3.0.0b8 → span_panel_api-3.0.0b10}/tests/test_redispatch_on_reconnect.py

@@ -31,7 +31,11 @@ from unittest.mock import patch
 import pytest
 
 from span_panel_api.exceptions import SpanPanelConnectionError, SpanPanelServerError
-from span_panel_api.mqtt.client import _REDISPATCH_RETRY_ATTEMPTS, SpanMqttClient
+from span_panel_api.mqtt.client import (
+    _REDISPATCH_RETRY_INITIAL_S,
+    _REDISPATCH_RETRY_MAX_S,
+    SpanMqttClient,
+)
 from span_panel_api.mqtt.models import MqttClientConfig
 
 from conftest import SERIAL
@@ -96,7 +100,7 @@ async def _panel_publishes_version(client: SpanMqttClient, version: str | None)
     # The refetch is scheduled rather than awaited, so the message callback can stay
     # synchronous. Let the loop drain it.
     # One turn per retry attempt, plus slack for the task itself.
-    for _ in range(_REDISPATCH_RETRY_ATTEMPTS + 4):
+    for _ in range(24):
         await asyncio.sleep(0)
 
 
@@ -210,13 +214,15 @@ async def test_http_lagging_the_broker_is_retried_not_abandoned() -> None:
 
 
 @pytest.mark.asyncio
-async def test_a_panel_that_never_serves_http_leaves_the_parser_alone() -> None:
-    """Bounded, and non-fatal when the bound is reached.
-
-    MQTT is up or this path would not be running, so tearing the connection down over
-    an unreachable HTTP endpoint would turn a degraded panel into a dead integration.
-    A stale parser reports missing data rather than wrong data, because the two
-    schemas share no topic shape.
+async def test_a_panel_that_is_not_serving_http_yet_leaves_the_parser_alone() -> None:
+    """Waiting must not disturb what is already working.
+
+    MQTT is up or this path would not be running, so tearing the connection down
+    over an HTTP endpoint that has not come up would turn a panel that is merely
+    booting into a dead integration. The parser stays as it is while the wait
+    runs — a stale parser reports missing data rather than wrong data, because
+    the two schemas share no topic shape — and the wait keeps going rather than
+    giving up, because nothing else will start it again.
     """
     client, _ = _client(None)
     before = client.adapter
@@ -231,8 +237,16 @@ async def test_a_panel_that_never_serves_http_leaves_the_parser_alone() -> None:
     ):
         await _panel_publishes_version(client, "1.0")
 
-    assert client.adapter is before
-    assert not client._redispatch_in_flight, "the in-flight guard must clear on failure"
+        assert client.adapter is before
+        assert client._redispatch_in_flight, (
+            "the guard is held for as long as the wait runs, so a second edge does not " "start a competing attempt"
+        )
+
+    # Cancelled directly rather than through `close()`, which this fixture's fake
+    # bridge cannot service. That the wait ends on cancellation is covered by
+    # `test_the_wait_ends_promptly_when_the_client_is_closed`.
+    for task in list(client._background_tasks):
+        task.cancel()
 
 
 @pytest.mark.asyncio
@@ -364,3 +378,105 @@ async def test_an_unexpected_failure_leaves_a_usable_message_rather_than_a_bare_
     assert client.adapter is before
     assert "Reload the integration" in caplog.text
     assert "something nobody predicted" in caplog.text
+
+
+@pytest.mark.asyncio
+async def test_the_backoff_reaches_a_steady_state_rather_than_growing() -> None:
+    """Once the panel is up, the wait to notice it must stay short.
+
+    Doubling without a ceiling would mean a panel that took a while to come back
+    was then ignored for longer than it took — minutes between attempts by the
+    time it is answering. The interval has to settle, so the worst case between
+    the panel being ready and this loop finding out is one interval however long
+    the wait has already run.
+
+    **Observed from the loop, not recomputed.** The first version of this test
+    calculated the backoff sequence itself and asserted on its own arithmetic,
+    which passes just as happily when the ceiling is removed from the code — the
+    same mistake as the window test it replaced. These are the sleeps the real
+    function performed.
+    """
+    client, _ = _client(None)
+    slept: list[float] = []
+    attempts = 0
+
+    def _ready_eventually(*_a: object, **_k: object) -> _Schema:
+        nonlocal attempts
+        attempts += 1
+        if attempts < 30:
+            raise SpanPanelServerError("Panel not ready: HTTP 502", 502)
+        return _Schema("1.0")
+
+    async def _record(seconds: float) -> None:
+        slept.append(seconds)
+
+    with (
+        patch("span_panel_api.mqtt.client.get_homie_schema", side_effect=_ready_eventually),
+        patch("span_panel_api.mqtt.client.asyncio.sleep", _record),
+    ):
+        assert await client._fetch_schema_with_retry() is not None
+
+    assert slept[0] == _REDISPATCH_RETRY_INITIAL_S, "it should start responsive"
+    assert max(slept) == _REDISPATCH_RETRY_MAX_S, "and never wait longer than the ceiling"
+    assert slept[-1] == _REDISPATCH_RETRY_MAX_S, "settling there rather than continuing to grow"
+    assert _REDISPATCH_RETRY_MAX_S <= 30.0, (
+        "a steady-state gap longer than half a minute is too long to leave a panel " "that is already answering"
+    )
+
+
+@pytest.mark.asyncio
+async def test_the_wait_does_not_end_on_its_own() -> None:
+    """There is no attempt count to exhaust, and that is the point.
+
+    Every bounded version of this was wrong, twice, for the same reason: the
+    bound was sized against a reboot somebody had measured and the next reboot
+    was not that reboot. Giving up has nothing to recommend it — the triggers for
+    another attempt are the reconnect edge and the retained message, and a panel
+    that finishes booting afterwards produces neither, so exhausting a bound
+    means stranded until a human reloads.
+    """
+    client, _ = _client(None)
+    attempts = 0
+
+    def _ready_far_later(*_a: object, **_k: object) -> _Schema:
+        nonlocal attempts
+        attempts += 1
+        if attempts < 40:  # well past any bound this ever had
+            raise SpanPanelServerError("Panel not ready: HTTP 502", 502)
+        return _Schema("1.0")
+
+    with (
+        patch("span_panel_api.mqtt.client.get_homie_schema", side_effect=_ready_far_later),
+        patch("span_panel_api.mqtt.client._REDISPATCH_RETRY_INITIAL_S", 0),
+        patch("span_panel_api.mqtt.client._REDISPATCH_RETRY_MAX_S", 0),
+    ):
+        assert await client._fetch_schema_with_retry() is not None
+
+    assert attempts == 40
+
+
+@pytest.mark.asyncio
+async def test_the_wait_ends_promptly_when_the_client_is_closed() -> None:
+    """Unbounded is only safe because cancellation is prompt.
+
+    `close()` cancels every background task, and the cancellation lands inside
+    the sleep. Without this, waiting forever would mean a Home Assistant
+    shutdown or a config-entry unload waiting with it.
+    """
+    client, _ = _client(None)
+
+    with (
+        patch(
+            "span_panel_api.mqtt.client.get_homie_schema",
+            side_effect=SpanPanelServerError("Panel not ready: HTTP 502", 502),
+        ),
+        patch("span_panel_api.mqtt.client._REDISPATCH_RETRY_INITIAL_S", 3600),
+        patch("span_panel_api.mqtt.client._REDISPATCH_RETRY_MAX_S", 3600),
+    ):
+        task = asyncio.create_task(client._fetch_schema_with_retry())
+        await asyncio.sleep(0)
+        task.cancel()
+        with pytest.raises(asyncio.CancelledError):
+            await task
+
+    assert task.cancelled()