updates2mqtt 1.7.3__tar.gz → 1.8.0__tar.gz

{updates2mqtt-1.7.3 → updates2mqtt-1.8.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: updates2mqtt
-Version: 1.7.3
+Version: 1.8.0
 Summary: System update and docker image notification and execution over MQTT
 Keywords: mqtt,docker,oci,container,updates,automation,home-assistant,homeassistant,selfhosting
 Author: jey burrows
@@ -72,7 +72,7 @@ Read the release notes, and optionally click *Update* to trigger a Docker *pull*
 
 Updates2MQTT perioidically checks for new versions of components being available, and publishes new version info to MQTT. HomeAssistant auto discovery is supported, so all updates can be seen in the same place as Home Assistant's own components and add-ins.
 
-Currently only Docker containers are supported, either via an image registry check (using either v1 Docker APIs or the OCI v2 API), or a git repo for source (see [Local Builds](local_builds.md)), with specific handling for Docker, Github Container Registry, Gitlab, Codeberg, Microsoft Container Registry and LinuxServer Registry, with adaptive behaviour to cope with most
+Currently only Docker containers are supported, either via an image registry check (using either v1 Docker APIs or the OCI v2 API), or a git repo for source (see [Local Builds](local_builds.md)), with specific handling for Docker, Github Container Registry, Gitlab, Codeberg, Microsoft Container Registry, Quay and LinuxServer Registry, with adaptive behaviour to cope with most
 others.  The design is modular, so other update sources can be added, at least for notification. The next anticipated is **apt** for Debian based systems.
 
 Components can also be updated, either automatically or triggered via MQTT, for example by hitting the *Install* button in the HomeAssistant update dialog. Icons and release notes can be specified for a better HA experience. See [Home Assistant Integration](home_assistant.md) for details.
@@ -91,6 +91,9 @@ or without Docker, using [uv](https://docs.astral.sh/uv/)
 export MQTT_HOST=192.168.1.1;export MQTT_USER=user1;export MQTT_PASS=user1;uv run --with updates2mqtt python -m updates2mqtt
 ```
 
+It also comes with a basic command line tool that will perform the analysis for a single running container, or fetch
+manifests, JSON blobs and lists of tags from remote registries (known to work with GitHub, GitLab, Codeberg, Quay, LSCR and Microsoft MCR).
+
 ## Release Support
 
 Presently only Docker containers are supported, although others are planned, probably with priority for `apt`.
@@ -128,7 +131,7 @@ restarter:
 While `updates2mqtt` will discover and monitor all containers running under the Docker daemon,
 there are some options to make to those containers to tune how it works.
 
-These happen by adding environment variables to the containers, typically inside an `.env`
+These happen by adding environment variables or docker labels to the containers, typically inside an `.env`
 file, or as `environment` options inside `docker-compose.yaml`.
 
 ### Automated updates
@@ -148,45 +151,6 @@ restarter:
 Automated updates can also apply to local builds, where a `git_repo_path` has been defined - if there are remote
 commits available to pull, then a `git pull`, `docker compose build` and `docker compose up` will be executed.
 
-### Environment Variables
-
-The following environment variables can be used to configure containers for `updates2mqtt`:
-
-| Env Var                    | Description                                                                                  | Default         |
-|----------------------------|----------------------------------------------------------------------------------------------|-----------------|
-| `UPD2MQTT_UPDATE`          | Update mode, either `Passive` or `Auto`. If `Auto`, updates will be installed automatically. | `Passive`       |
-| `UPD2MQTT_PICTURE`         | URL to an icon to use in Home Assistant.                                                     | Docker logo URL |
-| `UPD2MQTT_RELNOTES`        | URL to release notes for the package.                                                        |                 |
-| `UPD2MQTT_GIT_REPO_PATH`   | Relative path to a local git repo if the image is built locally.                             |                 |
-| `UPD2MQTT_IGNORE`          | If set to `True`, the container will be ignored by Updates2MQTT.                             | False           |
-                        |                 |
-| `UPD2MQTT_VERSION_POLICY` | Change how version derived from container label or image hash, `Version`,`Digest`,`Version_Digest` with default of `Auto`|
-| `UPD2MQTT_REGISTRY_TOKEN` | Access token for authentication to container distribution API, as alternative to making a call to `token` service |
-
-### Docker Labels
-
-Alternatively, use Docker labels
-
-| Label                          | Env Var                    |
-|--------------------------------|----------------------------|
-| `updates2mqtt.update`          | `UPD2MQTT_UPDATE`          |
-| `updates2mqtt.picture`         | `UPD2MQTT_PCITURE`         |
-| `updates2mqtt.relnotes`        | `UPD2MQTT_RELNOTES`        |
-| `updates2mqtt.git_repo_path`   | `UPD2MQTT_GIT_REPO_PATH`   |
-| `updates2mqtt.ignore`          | `UPD2MQTT_IGNORE`          |
-| `updates2mqtt.version_policy`  | `UPD2MQTT_VERSION_POLICY`  |
-| `updates2mqtt.registry_token`  | `UPD2MQTT_REGISTRY_TOKEN`  |
-
-
-
-```yaml title="Example Compose Snippet"
-restarter:
-    image: docker:cli
-    command: ["/bin/sh", "-c", "while true; do sleep 86400; docker restart mailserver; done"]
-    labels:
-        updates2mqtt.relnotes: https://component.my.com/release_notes
-```
-
 
 ## Related Projects
 
@@ -196,7 +160,7 @@ Other apps useful for self-hosting with the help of MQTT:
 
 Find more at [awesome-mqtt](https://github.com/rhizomatics/awesome-mqtt)
 
-For a more powerful Docker update manager, try [What's Up Docker](https://getwud.github.io/wud/)
+For a more powerful Docker focussed update manager, try [What's Up Docker](https://getwud.github.io/wud/)
 
 ## Development
 

{updates2mqtt-1.7.3 → updates2mqtt-1.8.0}/README.md

@@ -34,7 +34,7 @@ Read the release notes, and optionally click *Update* to trigger a Docker *pull*
 
 Updates2MQTT perioidically checks for new versions of components being available, and publishes new version info to MQTT. HomeAssistant auto discovery is supported, so all updates can be seen in the same place as Home Assistant's own components and add-ins.
 
-Currently only Docker containers are supported, either via an image registry check (using either v1 Docker APIs or the OCI v2 API), or a git repo for source (see [Local Builds](local_builds.md)), with specific handling for Docker, Github Container Registry, Gitlab, Codeberg, Microsoft Container Registry and LinuxServer Registry, with adaptive behaviour to cope with most
+Currently only Docker containers are supported, either via an image registry check (using either v1 Docker APIs or the OCI v2 API), or a git repo for source (see [Local Builds](local_builds.md)), with specific handling for Docker, Github Container Registry, Gitlab, Codeberg, Microsoft Container Registry, Quay and LinuxServer Registry, with adaptive behaviour to cope with most
 others.  The design is modular, so other update sources can be added, at least for notification. The next anticipated is **apt** for Debian based systems.
 
 Components can also be updated, either automatically or triggered via MQTT, for example by hitting the *Install* button in the HomeAssistant update dialog. Icons and release notes can be specified for a better HA experience. See [Home Assistant Integration](home_assistant.md) for details.
@@ -53,6 +53,9 @@ or without Docker, using [uv](https://docs.astral.sh/uv/)
 export MQTT_HOST=192.168.1.1;export MQTT_USER=user1;export MQTT_PASS=user1;uv run --with updates2mqtt python -m updates2mqtt
 ```
 
+It also comes with a basic command line tool that will perform the analysis for a single running container, or fetch
+manifests, JSON blobs and lists of tags from remote registries (known to work with GitHub, GitLab, Codeberg, Quay, LSCR and Microsoft MCR).
+
 ## Release Support
 
 Presently only Docker containers are supported, although others are planned, probably with priority for `apt`.
@@ -90,7 +93,7 @@ restarter:
 While `updates2mqtt` will discover and monitor all containers running under the Docker daemon,
 there are some options to make to those containers to tune how it works.
 
-These happen by adding environment variables to the containers, typically inside an `.env`
+These happen by adding environment variables or docker labels to the containers, typically inside an `.env`
 file, or as `environment` options inside `docker-compose.yaml`.
 
 ### Automated updates
@@ -110,45 +113,6 @@ restarter:
 Automated updates can also apply to local builds, where a `git_repo_path` has been defined - if there are remote
 commits available to pull, then a `git pull`, `docker compose build` and `docker compose up` will be executed.
 
-### Environment Variables
-
-The following environment variables can be used to configure containers for `updates2mqtt`:
-
-| Env Var                    | Description                                                                                  | Default         |
-|----------------------------|----------------------------------------------------------------------------------------------|-----------------|
-| `UPD2MQTT_UPDATE`          | Update mode, either `Passive` or `Auto`. If `Auto`, updates will be installed automatically. | `Passive`       |
-| `UPD2MQTT_PICTURE`         | URL to an icon to use in Home Assistant.                                                     | Docker logo URL |
-| `UPD2MQTT_RELNOTES`        | URL to release notes for the package.                                                        |                 |
-| `UPD2MQTT_GIT_REPO_PATH`   | Relative path to a local git repo if the image is built locally.                             |                 |
-| `UPD2MQTT_IGNORE`          | If set to `True`, the container will be ignored by Updates2MQTT.                             | False           |
-                        |                 |
-| `UPD2MQTT_VERSION_POLICY` | Change how version derived from container label or image hash, `Version`,`Digest`,`Version_Digest` with default of `Auto`|
-| `UPD2MQTT_REGISTRY_TOKEN` | Access token for authentication to container distribution API, as alternative to making a call to `token` service |
-
-### Docker Labels
-
-Alternatively, use Docker labels
-
-| Label                          | Env Var                    |
-|--------------------------------|----------------------------|
-| `updates2mqtt.update`          | `UPD2MQTT_UPDATE`          |
-| `updates2mqtt.picture`         | `UPD2MQTT_PCITURE`         |
-| `updates2mqtt.relnotes`        | `UPD2MQTT_RELNOTES`        |
-| `updates2mqtt.git_repo_path`   | `UPD2MQTT_GIT_REPO_PATH`   |
-| `updates2mqtt.ignore`          | `UPD2MQTT_IGNORE`          |
-| `updates2mqtt.version_policy`  | `UPD2MQTT_VERSION_POLICY`  |
-| `updates2mqtt.registry_token`  | `UPD2MQTT_REGISTRY_TOKEN`  |
-
-
-
-```yaml title="Example Compose Snippet"
-restarter:
-    image: docker:cli
-    command: ["/bin/sh", "-c", "while true; do sleep 86400; docker restart mailserver; done"]
-    labels:
-        updates2mqtt.relnotes: https://component.my.com/release_notes
-```
-
 
 ## Related Projects
 
@@ -158,7 +122,7 @@ Other apps useful for self-hosting with the help of MQTT:
 
 Find more at [awesome-mqtt](https://github.com/rhizomatics/awesome-mqtt)
 
-For a more powerful Docker update manager, try [What's Up Docker](https://getwud.github.io/wud/)
+For a more powerful Docker focussed update manager, try [What's Up Docker](https://getwud.github.io/wud/)
 
 ## Development
 

{updates2mqtt-1.7.3 → updates2mqtt-1.8.0}/pyproject.toml

@@ -7,7 +7,7 @@ authors = [
 ]
 
 requires-python = ">=3.13"
-version = "1.7.3"
+version = "1.8.0"
 license="Apache-2.0"
 keywords=["mqtt", "docker", "oci","container","updates", "automation","home-assistant","homeassistant","selfhosting"]
 
@@ -60,7 +60,8 @@ dev = [
     "pytest-subprocess>=1.5.3",
     "coverage",
     "icdiff",
-    "genbadge[all]"
+    "genbadge[all]",
+    "actionlint-py"
 ]
 docs=[
     "mkdocs",
@@ -74,7 +75,8 @@ docs=[
     "mkdocs-git-revision-date-localized-plugin",
     "mkdocs-meta-descriptions-plugin",
     "pngquant",
-    "mkdocs-mermaid2-plugin"
+    "mkdocs-mermaid2-plugin",
+    "mkdocs-htmlproofer-plugin"
 ]
 
 [build-system]

{updates2mqtt-1.7.3 → updates2mqtt-1.8.0}/src/updates2mqtt/app.py

@@ -56,7 +56,15 @@ class App:
         self.scan_count: int = 0
         self.last_scan: str | None = None
         if self.cfg.docker.enabled:
-            self.scanners.append(DockerProvider(self.cfg.docker, self.cfg.node, self.self_bounce))
+            self.scanners.append(
+                DockerProvider(
+                    self.cfg.docker,
+                    self.cfg.node,
+                    packages=self.cfg.packages,
+                    github_cfg=self.cfg.github,
+                    self_bounce=self.self_bounce,
+                )
+            )
         self.stopped = Event()
         self.healthcheck_topic = self.cfg.node.healthcheck.topic_template.format(node_name=self.cfg.node.name)
 
@@ -71,9 +79,6 @@ class App:
         session = uuid.uuid4().hex
         for scanner in self.scanners:
             slog = log.bind(source_type=scanner.source_type, session=session)
-            slog.info("Cleaning topics before scan")
-            if self.scan_count == 0:
-                await self.publisher.clean_topics(scanner, None, force=True)
             if self.stopped.is_set():
                 break
             slog.info("Scanning ...")
@@ -84,7 +89,7 @@ class App:
             if self.stopped.is_set():
                 slog.debug("Breaking scan loop on stopped event")
                 break
-            await self.publisher.clean_topics(scanner, session, force=False)
+            await self.publisher.clean_topics(scanner)
             self.scan_count += 1
             slog.info(f"Scan #{self.scan_count} complete")
         self.last_scan_timestamp = datetime.now(UTC).isoformat()

{updates2mqtt-1.7.3 → updates2mqtt-1.8.0}/src/updates2mqtt/cli.py

@@ -4,7 +4,7 @@ import structlog
 from omegaconf import DictConfig, OmegaConf
 from rich import print_json
 
-from updates2mqtt.config import DockerConfig, NodeConfig, RegistryConfig
+from updates2mqtt.config import DockerConfig, GitHubConfig, NodeConfig, RegistryConfig
 from updates2mqtt.helpers import Throttler
 from updates2mqtt.integrations.docker import DockerProvider
 from updates2mqtt.integrations.docker_enrich import (
@@ -24,18 +24,20 @@ log = structlog.get_logger()
 """
 Super simple CLI
 
-python updates2mqtt.cli container=frigate
+Command can be `container`,`tags`,`manifest` or `blob`
 
-python updates2mqtt.cli container=frigate api=docker_client log_level=DEBUG
+* `container=container-name`
+* `container=hash`
+* `tags=ghcr.io/
+* `blob=manifest=mcr.microsoft.com/dotnet/sdk:latest`
+* `tags=quay.io/linuxserver.io/babybuddy`
+* `blob=ghcr.io/blakeblackshear/frigate@sha256:759c36ee869e3e60258350a2e221eae1a4ba1018613e0334f1bc84eb09c4bbbc`
 
-ython3 updates2mqtt/cli.py blob=ghcr.io/homarr-labs/homarr@sha256:af79a3339de5ed8ef7f5a0186ff3deb86f40b213ba75249291f2f68aef082a25 | jq '.config.Labels'
+In addition, a `log_level=DEBUG` or other level can be added, `github_token` to try a personal access
+token for GitHub release info retrieval, or `api=docker_client` to use the older API (defaults to `api=OCI_V2`)
 
-python3 updates2mqtt/cli.py manifest=ghcr.io/blakeblackshear/frigate:stable
 
-python3 updates2mqtt/cli.py blob=ghcr.io/blakeblackshear/frigate@sha256:ef8d56a7d50b545af176e950ce328aec7f0b7bc5baebdca189fe661d97924980
-
-python3 updates2mqtt/cli.py manifest=ghcr.io/blakeblackshear/frigate@sha256:c68fd78fd3237c9ba81b5aa927f17b54f46705990f43b4b5d5596cfbbb626af4
-"""  # noqa: E501
+"""
 
 OCI_MANIFEST_TYPES: list[str] = [
     "application/vnd.oci.image.manifest.v1+json",
@@ -91,7 +93,9 @@ ALL_OCI_MEDIA_TYPES: list[str] = (
 )
 
 
-def dump_url(doc_type: str, img_ref: str) -> None:
+def dump_url(doc_type: str, img_ref: str, cli_conf: DictConfig) -> None:
+    structlog.configure(wrapper_class=structlog.make_filtering_bound_logger(cli_conf.get("log_level", "WARNING")))
+
     lookup = ContainerDistributionAPIVersionLookup(Throttler(), RegistryConfig())
     img_info = DockerImageInfo(img_ref)
     if not img_info.index_name or not img_info.name:
@@ -110,35 +114,49 @@ def dump_url(doc_type: str, img_ref: str) -> None:
             log.warning("No tag or digest found in %s", img_ref)
             return
         url = f"https://{api_host}/v2/{img_info.name}/manifests/{img_info.tag_or_digest}"
+    elif doc_type == "tags":
+        url = f"https://{api_host}/v2/{img_info.name}/tags/list"
     else:
         return
 
     token: str | None = lookup.fetch_token(img_info.index_name, img_info.name)
 
     response: Response | None = fetch_url(url, bearer_token=token, follow_redirects=True, response_type=ALL_OCI_MEDIA_TYPES)
-    if response:
+    if response and response.is_error:
+        log.warning(f"{response.status_code}: {url}")
+        log.warning(response.text)
+    elif response and response.is_success:
         log.debug(f"{response.status_code}: {url}")
         log.debug("HEADERS")
         for k, v in response.headers.items():
             log.debug(f"{k}: {v}")
         log.debug("CONTENTS")
+
         print_json(response.text)
 
 
 def main() -> None:
     # will be a proper cli someday
     cli_conf: DictConfig = OmegaConf.from_cli()
-    structlog.configure(wrapper_class=structlog.make_filtering_bound_logger(cli_conf.get("log_level", "WARNING")))
 
     if cli_conf.get("blob"):
-        dump_url("blob", cli_conf.get("blob"))
+        dump_url("blob", cli_conf.get("blob"), cli_conf)
     elif cli_conf.get("manifest"):
-        dump_url("manifest", cli_conf.get("manifest"))
+        dump_url("manifest", cli_conf.get("manifest"), cli_conf)
+    elif cli_conf.get("tags"):
+        dump_url("tags", cli_conf.get("tags"), cli_conf)
 
     else:
+        structlog.configure(wrapper_class=structlog.make_filtering_bound_logger(cli_conf.get("log_level", "INFO")))
+
         docker_scanner = DockerProvider(
-            DockerConfig(registry=RegistryConfig(api=cli_conf.get("api", "OCI_V2"))), NodeConfig(), None
+            DockerConfig(registry=RegistryConfig(api=cli_conf.get("api", "OCI_V2"))),
+            NodeConfig(),
+            packages={},
+            github_cfg=GitHubConfig(access_token=cli_conf.get("github_token")),
+            self_bounce=None,
         )
+        docker_scanner.initialize()
         discovery: Discovery | None = docker_scanner.rescan(
             Discovery(docker_scanner, cli_conf.get("container", "frigate"), "cli", "manual")
         )

{updates2mqtt-1.7.3 → updates2mqtt-1.8.0}/src/updates2mqtt/config.py

@@ -67,6 +67,11 @@ class MqttConfig:
     protocol: str = "${oc.env:MQTT_VERSION,3.11}"
 
 
+@dataclass
+class GitHubConfig:
+    access_token: str | None = None
+
+
 @dataclass
 class MetadataSourceConfig:
     enabled: bool = True
@@ -86,6 +91,24 @@ class VersionPolicy(StrEnum):
     VERSION_DIGEST = "VERSION_DIGEST"
 
 
+@dataclass
+class DockerPackageUpdateInfo:
+    image_name: str = MISSING  # untagged image ref
+
+
+@dataclass
+class PackageUpdateInfo:
+    docker: DockerPackageUpdateInfo | None = field(default_factory=DockerPackageUpdateInfo)
+    logo_url: str | None = None
+    release_notes_url: str | None = None
+    source_repo_url: str | None = None
+
+
+@dataclass
+class UpdateInfoConfig:
+    common_packages: dict[str, PackageUpdateInfo] = field(default_factory=lambda: {})
+
+
 @dataclass
 class DockerConfig:
     enabled: bool = True
@@ -149,25 +172,9 @@ class Config:
     mqtt: MqttConfig = field(default_factory=MqttConfig)  # pyright: ignore[reportArgumentType, reportCallIssue]
     homeassistant: HomeAssistantConfig = field(default_factory=HomeAssistantConfig)
     docker: DockerConfig = field(default_factory=DockerConfig)
+    github: GitHubConfig = field(default_factory=GitHubConfig)
     scan_interval: int = 60 * 60 * 3
-
-
-@dataclass
-class DockerPackageUpdateInfo:
-    image_name: str = MISSING  # untagged image ref
-
-
-@dataclass
-class PackageUpdateInfo:
-    docker: DockerPackageUpdateInfo | None = field(default_factory=DockerPackageUpdateInfo)
-    logo_url: str | None = None
-    release_notes_url: str | None = None
-    source_repo_url: str | None = None
-
-
-@dataclass
-class UpdateInfoConfig:
-    common_packages: dict[str, PackageUpdateInfo] = field(default_factory=lambda: {})
+    packages: dict[str, PackageUpdateInfo] = field(default_factory=dict)
 
 
 class IncompleteConfigException(BaseException):
@@ -188,13 +195,13 @@ def load_app_config(conf_file_path: Path, return_invalid: bool = False) -> Confi
             try:
                 log.debug(f"Creating config directory {conf_file_path.parent} if not already present")
                 conf_file_path.parent.mkdir(parents=True, exist_ok=True)
-            except Exception:
-                log.warning("Unable to create config directory", path=conf_file_path.parent)
+            except Exception as e:
+                log.warning("Unable to create config directory: %s", e, path=conf_file_path.parent)
         try:
             conf_file_path.write_text(OmegaConf.to_yaml(base_cfg))
             log.info(f"Auto-generated a new config file at {conf_file_path}")
-        except Exception:
-            log.warning("Unable to write config file", path=conf_file_path)
+        except Exception as e:
+            log.warning("Unable to write config file: %s", e, path=conf_file_path)
         cfg = base_cfg
     else:
         cfg = base_cfg

{updates2mqtt-1.7.3 → updates2mqtt-1.8.0}/src/updates2mqtt/helpers.py

@@ -145,8 +145,8 @@ class APIStats:
         """Log line friendly string summary"""
         return (
             f"fetches: {self.fetches}, cache ratio: {self.hit_ratio():.2%}, revalidated: {self.revalidated}, "
-            + f"errors: {', '.join(f'{status_code}:{fails}' for status_code, fails in self.failed.items())}, "
-            + f"oldest cache hit: {self.max_cache_age:.2f}, avg elapsed: {self.average_elapsed()}"
+            + f"errors: {', '.join(f'{status_code}:{fails}' for status_code, fails in self.failed.items()) or '0'}, "
+            + f"oldest cache hit: {self.max_cache_age:.2f}s, avg elapsed: {self.average_elapsed()}s"
         )
 
 
@@ -196,8 +196,9 @@ def fetch_url(
                 allow_stale=allow_stale,
             )
         )
+        log_headers: list[tuple[str, str]] = [h for h in headers if len(h) > 1 and h[0] != "Authorization"]
         with SyncCacheClient(headers=headers, follow_redirects=follow_redirects, policy=cache_policy) as client:
-            log.debug(f"Fetching URL {url}, redirects={follow_redirects}, headers={headers}, cache_ttl={cache_ttl}")
+            log.debug(f"Fetching URL {url}, redirects={follow_redirects}, headers={log_headers}, cache_ttl={cache_ttl}")
             response: Response = client.request(method=method, url=url, extensions={"hishel_ttl": cache_ttl})
             cache_metadata: CacheMetadata = CacheMetadata(response)
             if not response.is_success:
@@ -221,6 +222,45 @@ def fetch_url(
     return None
 
 
-def validate_url(url: str, cache_ttl: int = 300) -> bool:
+def validate_url(url: str, cache_ttl: int = 1500) -> bool:
     response: Response | None = fetch_url(url, method="HEAD", cache_ttl=cache_ttl, follow_redirects=True)
     return response is not None and response.status_code != 404
+
+
+def sanitize_name(name: str, replacement: str = "_", max_len: int = 64) -> str:
+    """Strict sanitization that removes/replaces common problematic characters for MQTT or HA
+
+    - Replaces spaces with underscores
+    - Removes control characters
+    - Ensures alphanumeric safety for broader compatibility
+
+    Args:
+        name: The topic component string to sanitize
+        replacement: Character to replace invalid characters with (default: "_")
+        max_len: Largest acceptable name size
+
+    Returns:
+        Sanitized topic string safe for most MQTT brokers
+
+    """
+    if not name:
+        raise ValueError("Name cannot be empty")
+    orig_name: str = name
+    name = re.sub(r"[^A-Za-z0-9_\-\.]+", replacement, name)
+
+    # Replace multiple consecutive replacement chars with single one
+    if replacement:
+        pattern = re.escape(replacement) + "+"
+        name = re.sub(pattern, replacement, name)
+
+    # Trim to max length
+    topic_bytes = name.encode("utf-8")
+    if len(topic_bytes) > max_len:
+        name = topic_bytes[:max_len].decode("utf-8", errors="ignore")
+
+    if not name:
+        raise ValueError("Topic became empty after sanitization")
+    if name != orig_name:
+        log.info("Component name %s changed to %s for MQTT/HA compatibility", orig_name, name)
+
+    return name

{updates2mqtt-1.7.3 → updates2mqtt-1.8.0}/src/updates2mqtt/integrations/docker.py

@@ -19,6 +19,7 @@ from updates2mqtt.config import (
     UNKNOWN_VERSION,
     VERSION_RE,
     DockerConfig,
+    GitHubConfig,
     NodeConfig,
     PackageUpdateInfo,
     PublishPolicy,
@@ -127,6 +128,8 @@ class DockerProvider(ReleaseProvider):
         self,
         cfg: DockerConfig,
         node_cfg: NodeConfig,
+        packages: dict[str, PackageUpdateInfo] | None = None,
+        github_cfg: GitHubConfig | None = None,
         self_bounce: Event | None = None,
     ) -> None:
         super().__init__(node_cfg, "docker")
@@ -136,8 +139,9 @@ class DockerProvider(ReleaseProvider):
         # TODO: refresh discovered packages periodically
         self.throttler = Throttler(self.cfg.default_api_backoff, self.log, self.stopped)
         self.self_bounce: Event | None = self_bounce
+
         self.pkg_enrichers: list[PackageEnricher] = [
-            CommonPackageEnricher(self.cfg),
+            CommonPackageEnricher(self.cfg, packages),
             LinuxServerIOPackageEnricher(self.cfg),
             DefaultPackageEnricher(self.cfg),
         ]
@@ -145,12 +149,13 @@ class DockerProvider(ReleaseProvider):
             self.client, self.throttler, self.cfg.registry, self.cfg.default_api_backoff
         )
         self.registry_image_lookup = ContainerDistributionAPIVersionLookup(self.throttler, self.cfg.registry)
-        self.release_enricher = SourceReleaseEnricher()
+        self.release_enricher = SourceReleaseEnricher(github_cfg)
         self.local_info_builder = LocalContainerInfo()
 
     def initialize(self) -> None:
         for enricher in self.pkg_enrichers:
             enricher.initialize()
+        self.log.debug("Docker provider initialized")
 
     def update(self, discovery: Discovery) -> bool:
         logger: Any = self.log.bind(container=discovery.name, action="update")
@@ -267,11 +272,11 @@ class DockerProvider(ReleaseProvider):
         )
 
     def rescan(self, discovery: Discovery) -> Discovery | None:
-        logger = self.log.bind(container=discovery.name, action="rescan")
+        logger: Any = self.log.bind(container=discovery.name, action="rescan")
         try:
             c: Container = self.client.containers.get(discovery.name)
             if c:
-                rediscovery = self.analyze(c, discovery.session, previous_discovery=discovery)
+                rediscovery: Discovery | None = self.analyze(c, discovery.session, previous_discovery=discovery)
                 if rediscovery and not rediscovery.throttled:
                     self.discoveries[rediscovery.name] = rediscovery
                     return rediscovery

{updates2mqtt-1.7.3 → updates2mqtt-1.8.0}/src/updates2mqtt/integrations/docker_enrich.py

@@ -23,21 +23,32 @@ from updates2mqtt.config import (
     PKG_INFO_FILE,
     DockerConfig,
     DockerPackageUpdateInfo,
+    GitHubConfig,
     PackageUpdateInfo,
     RegistryConfig,
     UpdateInfoConfig,
 )
 
-log = structlog.get_logger()
+log: Any = structlog.get_logger()
 
 SOURCE_PLATFORM_GITHUB = "GitHub"
 SOURCE_PLATFORM_CODEBERG = "CodeBerg"
-SOURCE_PLATFORMS = {SOURCE_PLATFORM_GITHUB: r"https://github.com/.*"}
+SOURCE_PLATFORM_GITLAB = "GitLab"
+SOURCE_PLATFORMS = {
+    SOURCE_PLATFORM_GITHUB: r"https://github.com/.*",
+    SOURCE_PLATFORM_GITLAB: r"https://gitlab.com/.*",
+    SOURCE_PLATFORM_CODEBERG: r"https://codeberg.org/.*",
+}
 DIFF_URL_TEMPLATES = {
     SOURCE_PLATFORM_GITHUB: "{repo}/commit/{revision}",
 }
-RELEASE_URL_TEMPLATES = {SOURCE_PLATFORM_GITHUB: "{repo}/releases/tag/{version}"}
-UNKNOWN_RELEASE_URL_TEMPLATES = {SOURCE_PLATFORM_GITHUB: "{repo}/releases"}
+RELEASE_URL_TEMPLATES = {
+    SOURCE_PLATFORM_GITHUB: "{repo}/releases/tag/{version}",
+}
+UNKNOWN_RELEASE_URL_TEMPLATES = {
+    SOURCE_PLATFORM_GITHUB: "{repo}/releases",
+    SOURCE_PLATFORM_GITLAB: "{repo}/container_registry",
+}
 MISSING_VAL = "**MISSING**"
 UNKNOWN_REGISTRY = "**UNKNOWN_REGISTRY**"
 
@@ -45,18 +56,27 @@ HEADER_DOCKER_DIGEST = "docker-content-digest"
 HEADER_DOCKER_API = "docker-distribution-api-version"
 
 TOKEN_URL_TEMPLATE = "https://{auth_host}/token?scope=repository:{image_name}:pull&service={service}"  # noqa: S105 # nosec
-REGISTRIES = {
-    # registry: (auth_host, api_host, service, url_template)
-    "docker.io": ("auth.docker.io", "registry-1.docker.io", "registry.docker.io", TOKEN_URL_TEMPLATE),
-    "mcr.microsoft.com": (None, "mcr.microsoft.com", "mcr.microsoft.com", TOKEN_URL_TEMPLATE),
-    "ghcr.io": ("ghcr.io", "ghcr.io", "ghcr.io", TOKEN_URL_TEMPLATE),
-    "lscr.io": ("ghcr.io", "lscr.io", "ghcr.io", TOKEN_URL_TEMPLATE),
-    "codeberg.org": ("codeberg.org", "codeberg.org", "container_registry", TOKEN_URL_TEMPLATE),
+
+REGISTRIES: dict[str, tuple[str | None, str, str, str | None, str | None]] = {
+    # registry: (auth_host, api_host, service, url_template, repo_template)
+    "docker.io": ("auth.docker.io", "registry-1.docker.io", "registry.docker.io", TOKEN_URL_TEMPLATE, None),
+    "mcr.microsoft.com": (None, "mcr.microsoft.com", "mcr.microsoft.com", None, None),
+    "quay.io": (None, "quay.io", "quay.io", TOKEN_URL_TEMPLATE, None),
+    "ghcr.io": ("ghcr.io", "ghcr.io", "ghcr.io", TOKEN_URL_TEMPLATE, "https://github.com/{image_name}"),
+    "lscr.io": ("ghcr.io", "lscr.io", "ghcr.io", TOKEN_URL_TEMPLATE, None),
+    "codeberg.org": (
+        "codeberg.org",
+        "codeberg.org",
+        "container_registry",
+        TOKEN_URL_TEMPLATE,
+        "https://codeberg.org/{image_name}",
+    ),
     "registry.gitlab.com": (
         "www.gitlab.com",
         "registry.gitlab.com",
         "container_registry",
         "https://{auth_host}/jwt/auth?service={service}&scope=repository:{image_name}:pull&offline_token=true&client_id=docker",
+        "https://gitlab.com/{image_name}",
     ),
 }
 
@@ -190,7 +210,8 @@ class DockerImageInfo(DiscoveryArtefactDetail):
                 digest = digest.split(":")[1] if ":" in digest else digest  # remove digest type prefix
                 return digest[0:12]
             return digest
-        except Exception:
+        except Exception as e:
+            log.warning("Unable to condense digest %s: %s", digest, e)
             return None
 
     def reuse(self) -> "DockerImageInfo":
@@ -328,8 +349,8 @@ class LocalContainerInfo:
 
 
 class PackageEnricher:
-    def __init__(self, docker_cfg: DockerConfig) -> None:
-        self.pkgs: dict[str, PackageUpdateInfo] = {}
+    def __init__(self, docker_cfg: DockerConfig, packages: dict[str, PackageUpdateInfo] | None = None) -> None:
+        self.pkgs: dict[str, PackageUpdateInfo] = packages or {}
         self.cfg: DockerConfig = docker_cfg
         self.log: Any = structlog.get_logger().bind(integration="docker")
 
@@ -371,18 +392,18 @@ class DefaultPackageEnricher(PackageEnricher):
 class CommonPackageEnricher(PackageEnricher):
     def initialize(self) -> None:
         if PKG_INFO_FILE.exists():
-            log.debug("Loading common package update info", path=PKG_INFO_FILE)
+            self.log.debug("Loading common package update info", path=PKG_INFO_FILE)
             cfg = OmegaConf.load(PKG_INFO_FILE)
         else:
-            log.warn("No common package update info found", path=PKG_INFO_FILE)
+            self.log.warn("No common package update info found", path=PKG_INFO_FILE)
             cfg = OmegaConf.structured(UpdateInfoConfig)
         try:
             # omegaconf broken-ness on optional fields and converting to backclasses
             self.pkgs: dict[str, PackageUpdateInfo] = {
-                pkg: PackageUpdateInfo(**pkg_cfg) for pkg, pkg_cfg in cfg.common_packages.items()
+                pkg: PackageUpdateInfo(**pkg_cfg) for pkg, pkg_cfg in cfg.common_packages.items() if pkg not in self.pkgs
             }
         except (MissingMandatoryValue, ValidationError) as e:
-            log.error("Configuration error %s", e, path=PKG_INFO_FILE.as_posix())
+            self.log.error("Configuration error %s", e, path=PKG_INFO_FILE.as_posix())
             raise
 
 
@@ -392,7 +413,7 @@ class LinuxServerIOPackageEnricher(PackageEnricher):
         if cfg is None or not cfg.enabled:
             return
 
-        log.debug(f"Fetching linuxserver.io metadata from API, cache_ttl={cfg.cache_ttl}")
+        self.log.debug(f"Fetching linuxserver.io metadata from API, cache_ttl={cfg.cache_ttl}")
         response: Response | None = fetch_url(
             "https://api.linuxserver.io/api/v1/images?include_config=false&include_deprecated=false",
             cache_ttl=cfg.cache_ttl,
@@ -413,26 +434,38 @@ class LinuxServerIOPackageEnricher(PackageEnricher):
                     release_notes_url=f"{repo['github_url']}/releases",
                 )
                 added += 1
-                log.debug("Added linuxserver.io package", pkg=image_name)
-        log.info(f"Added {added} linuxserver.io package details")
+        self.log.info(f"Added {added} linuxserver.io package details")
 
 
 class SourceReleaseEnricher:
-    def __init__(self) -> None:
+    def __init__(self, gh_cfg: GitHubConfig | None = None) -> None:
         self.log: Any = structlog.get_logger().bind(integration="docker")
+        self.gh_cfg: GitHubConfig | None = gh_cfg
 
     def enrich(
         self, registry_info: DockerImageInfo, source_repo_url: str | None = None, notes_url: str | None = None
     ) -> ReleaseDetail | None:
-        if not registry_info.annotations and not source_repo_url and not notes_url:
-            return None
-
         detail = ReleaseDetail()
 
         detail.notes_url = notes_url
         detail.version = registry_info.annotations.get("org.opencontainers.image.version")
         detail.revision = registry_info.annotations.get("org.opencontainers.image.revision")
-        detail.source_url = registry_info.annotations.get("org.opencontainers.image.source") or source_repo_url
+        # explicit source_repo_url overrides container, e.g. where container source is only the docker wrapper
+        detail.source_url = source_repo_url or registry_info.annotations.get("org.opencontainers.image.source")
+
+        if detail.source_url is None and registry_info is not None and registry_info.index_name is not None:
+            registry_config: tuple[str | None, str, str, str | None, str | None] | None = REGISTRIES.get(
+                registry_info.index_name
+            )
+            repo_template: str | None = registry_config[4] if registry_config else None
+            if repo_template:
+                source_url = repo_template.format(image_name=registry_info.name)
+                if validate_url(source_url, cache_ttl=86400):
+                    detail.source_url = source_url
+                    self.log.info("Implied source from registry: %s", detail.source_url)
+
+        if detail.source_url is None and detail.notes_url is None and detail.revision is None and detail.version is None:
+            return None
 
         if detail.source_url and "#" in detail.source_url:
             detail.source_repo_url = detail.source_url.split("#", 1)[0]
@@ -452,24 +485,60 @@ class SourceReleaseEnricher:
             "source": detail.source_url or MISSING_VAL,
         }
 
-        diff_url: str | None = DIFF_URL_TEMPLATES[detail.source_platform].format(**template_vars)
-        if diff_url and MISSING_VAL not in diff_url and validate_url(diff_url):
+        diff_url_template: str | None = DIFF_URL_TEMPLATES.get(detail.source_platform)
+        diff_url: str | None = diff_url_template.format(**template_vars) if diff_url_template else None
+        if diff_url and MISSING_VAL not in diff_url and validate_url(diff_url, cache_ttl=3600):
             detail.diff_url = diff_url
         else:
             diff_url = None
 
-        if detail.notes_url is None:
-            detail.notes_url = RELEASE_URL_TEMPLATES[detail.source_platform].format(**template_vars)
-
-            if MISSING_VAL in detail.notes_url or not validate_url(detail.notes_url):
-                detail.notes_url = UNKNOWN_RELEASE_URL_TEMPLATES[detail.source_platform].format(**template_vars)
-                if MISSING_VAL in detail.notes_url or not validate_url(detail.notes_url):
-                    detail.notes_url = None
-
-        if detail.source_platform == SOURCE_PLATFORM_GITHUB and detail.source_repo_url:
+        if detail.notes_url is None and detail.source_platform in RELEASE_URL_TEMPLATES:
+            platform_notes_url: str | None = RELEASE_URL_TEMPLATES[detail.source_platform].format(**template_vars)
+            if (
+                platform_notes_url
+                and MISSING_VAL not in platform_notes_url
+                and validate_url(platform_notes_url, cache_ttl=86400)
+            ):
+                self.log.debug("Setting default known release notes url: %s", platform_notes_url)
+                detail.notes_url = platform_notes_url
+
+        if detail.notes_url is None and detail.source_platform in UNKNOWN_RELEASE_URL_TEMPLATES:
+            platform_notes_url = UNKNOWN_RELEASE_URL_TEMPLATES[detail.source_platform].format(**template_vars)
+            if (
+                platform_notes_url
+                and MISSING_VAL not in platform_notes_url
+                and validate_url(platform_notes_url, cache_ttl=86400)
+            ):
+                self.log.debug("Setting default unknown release notes url: %s", platform_notes_url)
+                detail.notes_url = platform_notes_url
+
+        if detail.source_platform == SOURCE_PLATFORM_GITHUB and detail.source_repo_url and detail.version is not None:
+            access_token: str | None = self.gh_cfg.access_token if self.gh_cfg else None
+            if access_token:
+                self.log.debug("Using configured bearer token (%s chars) for GitHub API", len(access_token))
             base_api = detail.source_repo_url.replace("https://github.com", "https://api.github.com/repos")
 
-            api_response: Response | None = fetch_url(f"{base_api}/releases/tags/{detail.version}")
+            api_response: Response | None = fetch_url(
+                f"{base_api}/releases/tags/{detail.version}", bearer_token=access_token, allow_stale=True
+            )
+            if api_response and api_response.status_code == 404:
+                # possible that source version doesn't match release gag
+                alt_api_response: Response | None = fetch_url(f"{base_api}/releases/latest", bearer_token=access_token)
+                if alt_api_response and alt_api_response.is_success:
+                    alt_api_results = httpx_json_content(alt_api_response, {})
+                    if alt_api_results and re.fullmatch(f"(V|v|r|R)?{detail.version}", alt_api_results.get("tag_name")):
+                        self.log.info(
+                            f"Matched {registry_info.name} {detail.version} to latest release {alt_api_results['tag_name']}"
+                        )
+                        api_response = alt_api_response
+                    elif alt_api_results:
+                        self.log.debug(
+                            "Failed to match latest release for %s, found tag %s for name %s",
+                            detail.version,
+                            alt_api_results.get("tag_name"),
+                            alt_api_results.get("name"),
+                        )
+
             if api_response and api_response.is_success:
                 api_results: Any = httpx_json_content(api_response, {})
                 detail.summary = api_results.get("body")  # ty:ignore[possibly-missing-attribute]
@@ -523,14 +592,18 @@ class ContainerDistributionAPIVersionLookup(VersionLookup):
         self.api_stats = APIStatsCounter()
 
     def fetch_token(self, registry: str, image_name: str) -> str | None:
-        default_host: tuple[str, str, str, str] = (registry, registry, registry, TOKEN_URL_TEMPLATE)
+        default_host: tuple[str, str, str, str, None] = (registry, registry, registry, TOKEN_URL_TEMPLATE, None)
         auth_host: str | None = REGISTRIES.get(registry, default_host)[0]
         if auth_host is None:
             return None
 
         service: str = REGISTRIES.get(registry, default_host)[2]
-        url_template: str = REGISTRIES.get(registry, default_host)[3]
-        auth_url: str = url_template.format(auth_host=auth_host, image_name=image_name, service=service)
+        url_template: str | None = REGISTRIES.get(registry, default_host)[3]
+        auth_url: str | None = (
+            url_template.format(auth_host=auth_host, image_name=image_name, service=service) if url_template else None
+        )
+        if auth_url is None:
+            return None
         response: Response | None = fetch_url(
             auth_url, cache_ttl=self.cfg.token_cache_ttl, follow_redirects=True, api_stats_counter=self.api_stats
         )
@@ -737,7 +810,7 @@ class ContainerDistributionAPIVersionLookup(VersionLookup):
                     if index_digest:
                         result.image_digest = index_digest
                         result.short_digest = result.condense_digest(index_digest)
-                        log.debug("Setting %s image digest %s", result.name, result.short_digest)
+                        self.log.debug("Setting %s image digest %s", result.name, result.short_digest)
 
                     digest: str | None = m.get("digest")
                     media_type = m.get("mediaType")
@@ -757,7 +830,7 @@ class ContainerDistributionAPIVersionLookup(VersionLookup):
                             self.log.warning("Empty digest for %s %s %s", api_host, digest, media_type)
                         else:
                             result.repo_digest = result.condense_digest(digest, short=False)
-                            log.debug("Setting %s repo digest: %s", result.name, result.repo_digest)
+                            self.log.debug("Setting %s repo digest: %s", result.name, result.repo_digest)
 
                         if manifest.get("annotations"):
                             result.annotations.update(manifest.get("annotations", {}))

{updates2mqtt-1.7.3 → updates2mqtt-1.8.0}/src/updates2mqtt/model.py

@@ -8,7 +8,7 @@ from typing import Any
 import structlog
 
 from updates2mqtt.config import NodeConfig, PublishPolicy, UpdatePolicy, VersionPolicy
-from updates2mqtt.helpers import timestamp
+from updates2mqtt.helpers import sanitize_name, timestamp
 
 
 class DiscoveryArtefactDetail:
@@ -59,6 +59,10 @@ class ReleaseDetail:
             "net_score": str(self.net_score) if self.net_score is not None else None,
         }
 
+    def __str__(self) -> str:
+        """Log friendly"""
+        return ",".join(f"{k}:{v}" for k, v in self.as_dict().items())
+
 
 class Discovery:
     """Discovered component from a scan"""
@@ -94,7 +98,7 @@ class Discovery:
         self.provider: ReleaseProvider = provider
         self.source_type: str = provider.source_type
         self.session: str = session
-        self.name: str = name
+        self.name: str = sanitize_name(name)
         self.node: str = node
         self.entity_picture_url: str | None = entity_picture_url
         self.current_version: str | None = current_version

{updates2mqtt-1.7.3 → updates2mqtt-1.8.0}/src/updates2mqtt/mqtt.py

@@ -1,5 +1,6 @@
 import asyncio
 import json
+import re
 import time
 from collections.abc import Callable
 from dataclasses import dataclass, field
@@ -21,6 +22,8 @@ from .hass_formatter import hass_format_config, hass_format_state
 
 log = structlog.get_logger()
 
+MQTT_NAME = r"[A-Za-z0-9_\-\.]+"
+
 
 @dataclass
 class LocalMessage:
@@ -34,6 +37,7 @@ class MqttPublisher:
         self.node_cfg: NodeConfig = node_cfg
         self.hass_cfg: HomeAssistantConfig = hass_cfg
         self.providers_by_topic: dict[str, ReleaseProvider] = {}
+        self.providers_by_type: dict[str, ReleaseProvider] = {}
         self.event_loop: asyncio.AbstractEventLoop | None = None
         self.client: mqtt.Client | None = None
         self.fatal_failure = Event()
@@ -123,69 +127,60 @@ class MqttPublisher:
         else:
             self.log.warning("Disconnect failure from broker", result_code=rc)
 
-    async def clean_topics(
-        self, provider: ReleaseProvider, last_scan_session: str | None, wait_time: int = 5, force: bool = False
-    ) -> None:
+    async def clean_topics(self, provider: ReleaseProvider, wait_time: int = 5, max_time: int = 120) -> None:
         logger = self.log.bind(action="clean")
         if self.fatal_failure.is_set():
             return
-        logger.info("Starting clean cycle")
-        cleaner = mqtt.Client(
-            callback_api_version=CallbackAPIVersion.VERSION1,
-            client_id=f"updates2mqtt_clean_{self.node_cfg.name}",
-            clean_session=True,
-        )
-        results = {"cleaned": 0, "handled": 0, "discovered": 0, "last_timestamp": time.time()}
-        cleaner.username_pw_set(self.cfg.user, password=self.cfg.password)
-        cleaner.connect(host=self.cfg.host, port=self.cfg.port, keepalive=60)
-        prefixes = [
-            f"{self.hass_cfg.discovery.prefix}/update/{self.node_cfg.name}_{provider.source_type}_",
-            f"{self.cfg.topic_root}/{self.node_cfg.name}/{provider.source_type}/",
-        ]
-
-        def cleanup(_client: mqtt.Client, _userdata: Any, msg: mqtt.MQTTMessage) -> None:
-            if msg.retain and any(msg.topic.startswith(prefix) for prefix in prefixes):
-                session = None
+        try:
+            logger.info("Starting clean cycle, max time: %s", max_time)
+            cutoff_time: float = time.time() + max_time
+            cleaner = mqtt.Client(
+                callback_api_version=CallbackAPIVersion.VERSION1,
+                client_id=f"updates2mqtt_clean_{self.node_cfg.name}",
+                clean_session=True,
+            )
+            results = {"cleaned": 0, "matched": 0, "discovered": 0, "last_timestamp": time.time()}
+            cleaner.username_pw_set(self.cfg.user, password=self.cfg.password)
+            cleaner.connect(host=self.cfg.host, port=self.cfg.port, keepalive=60)
+
+            def cleanup(_client: mqtt.Client, _userdata: Any, msg: mqtt.MQTTMessage) -> None:
+                discovery: Discovery | None = None
+                if msg.topic.startswith(
+                    f"{self.hass_cfg.discovery.prefix}/update/{self.node_cfg.name}_{provider.source_type}_"
+                ):
+                    discovery = self.reverse_config_topic(msg.topic, provider.source_type)
+                elif msg.topic.startswith(
+                    f"{self.cfg.topic_root}/{self.node_cfg.name}/{provider.source_type}/"
+                ) and msg.topic.endswith("/state"):
+                    discovery = self.reverse_state_topic(msg.topic, provider.source_type)
+                elif msg.topic.startswith(f"{self.cfg.topic_root}/{self.node_cfg.name}/{provider.source_type}/"):
+                    discovery = self.reverse_general_topic(msg.topic, provider.source_type)
+                else:
+                    logger.debug("Ignoring other topic ", topic=msg.topic)
+                    return
+
                 results["discovered"] += 1
-                try:
-                    payload = self.safe_json_decode(msg.payload)
-                    session = payload.get("source_session")
-                except Exception as e:
-                    log.warn(
-                        "Unable to handle payload for %s: %s",
-                        msg.topic,
-                        e,
-                        exc_info=1,
-                    )
-                results["handled"] += 1
+                if discovery is not None:
+                    results["matched"] += 1
                 results["last_timestamp"] = time.time()
-                if session is not None and last_scan_session is not None and session != last_scan_session:
-                    log.debug("Removing stale msg", topic=msg.topic, session=session)
-                    cleaner.publish(msg.topic, "", retain=True)
-                    results["cleaned"] += 1
-                elif session is None and force:
-                    log.debug("Removing untrackable msg", topic=msg.topic)
+                if discovery is None:
+                    logger.debug("Removing unknown discovery", topic=msg.topic)
                     cleaner.publish(msg.topic, "", retain=True)
                     results["cleaned"] += 1
-                else:
-                    log.debug(
-                        "Retaining topic with current session: %s",
-                        msg.topic,
-                    )
-            else:
-                log.debug("Skipping clean of %s", msg.topic)
 
-        cleaner.on_message = cleanup
-        options = paho.mqtt.subscribeoptions.SubscribeOptions(noLocal=True)
-        cleaner.subscribe(f"{self.hass_cfg.discovery.prefix}/update/#", options=options)
-        cleaner.subscribe(f"{self.cfg.topic_root}/{self.node_cfg.name}/{provider.source_type}/#", options=options)
+            cleaner.on_message = cleanup
+            options = paho.mqtt.subscribeoptions.SubscribeOptions(noLocal=True)
+            cleaner.subscribe(f"{self.hass_cfg.discovery.prefix}/update/#", options=options)
+            cleaner.subscribe(f"{self.cfg.topic_root}/{self.node_cfg.name}/{provider.source_type}/#", options=options)
 
-        while time.time() - results["last_timestamp"] <= wait_time:
-            cleaner.loop(0.5)
+            while time.time() - results["last_timestamp"] <= wait_time and time.time() <= cutoff_time:
+                cleaner.loop(0.5)
 
-        log.info(
-            f"Clean completed, discovered:{results['discovered']}, handled:{results['handled']}, cleaned:{results['cleaned']}"
-        )
+            logger.info(
+                f"Cleaned - discovered:{results['discovered']}, matched:{results['matched']}, cleaned:{results['cleaned']}"
+            )
+        except Exception as e:
+            logger.error("Cleaning topics of stale entries failed: %s", e)
 
     def safe_json_decode(self, jsonish: str | bytes | None) -> dict:
         if jsonish is None:
@@ -233,7 +228,7 @@ class MqttPublisher:
                     source_type,
                     comp_name,
                 )
-                updated = provider.command(comp_name, command, on_update_start, on_update_end)
+                updated: bool = provider.command(comp_name, command, on_update_start, on_update_end)
                 discovery = provider.resolve(comp_name)
                 if updated and discovery:
                     if discovery.publish_policy == PublishPolicy.HOMEASSISTANT and self.hass_cfg.discovery.enabled:
@@ -305,12 +300,48 @@ class MqttPublisher:
         prefix = self.hass_cfg.discovery.prefix
         return f"{prefix}/update/{self.node_cfg.name}_{discovery.source_type}_{discovery.name}/update/config"
 
+    def reverse_config_topic(self, topic: str, source_type: str) -> Discovery | None:
+        match = re.fullmatch(
+            f"{self.hass_cfg.discovery.prefix}/update/{self.node_cfg.name}_{source_type}_({MQTT_NAME})/update/config",
+            topic,
+        )
+        if match and len(match.groups()) == 1:
+            discovery_name: str = match.group(1)
+            if discovery_name in self.providers_by_type[source_type].discoveries:
+                return self.providers_by_type[source_type].discoveries[discovery_name]
+
+        self.log.debug("MQTT CONFIG no match for %s", topic)
+        return None
+
     def state_topic(self, discovery: Discovery) -> str:
         return f"{self.cfg.topic_root}/{self.node_cfg.name}/{discovery.source_type}/{discovery.name}/state"
 
+    def reverse_state_topic(self, topic: str, source_type: str) -> Discovery | None:
+        match = re.fullmatch(
+            f"{self.cfg.topic_root}/{self.node_cfg.name}/{source_type}/({MQTT_NAME})/state",
+            topic,
+        )
+        if match and len(match.groups()) == 1:
+            discovery_name: str = match.group(1)
+            if discovery_name in self.providers_by_type[source_type].discoveries:
+                return self.providers_by_type[source_type].discoveries[discovery_name]
+
+        self.log.debug("MQTT STATE no match for %s", topic)
+        return None
+
     def general_topic(self, discovery: Discovery) -> str:
         return f"{self.cfg.topic_root}/{self.node_cfg.name}/{discovery.source_type}/{discovery.name}"
 
+    def reverse_general_topic(self, topic: str, source_type: str) -> Discovery | None:
+        match = re.fullmatch(f"{self.cfg.topic_root}/{self.node_cfg.name}/{source_type}/({MQTT_NAME})", topic)
+        if match and len(match.groups()) == 1:
+            discovery_name: str = match.group(1)
+            if discovery_name in self.providers_by_type[source_type].discoveries:
+                return self.providers_by_type[source_type].discoveries[discovery_name]
+
+        self.log.debug("MQTT ATTR no match for %s", topic)
+        return None
+
     def command_topic(self, provider: ReleaseProvider) -> str:
         return f"{self.cfg.topic_root}/{self.node_cfg.name}/{provider.source_type}"
 
@@ -363,6 +394,7 @@ class MqttPublisher:
         else:
             self.log.info("Handler subscribing", topic=topic)
             self.providers_by_topic[topic] = provider
+            self.providers_by_type[provider.source_type] = provider
             self.client.subscribe(topic)
         return topic