rowmonitor-airflow 0.1.0__tar.gz

rowmonitor_airflow-0.1.0/LICENSE

@@ -0,0 +1,4 @@
+Copyright (c) 2026 Row Monitor (Eklavyai). All rights reserved.
+
+This package is proprietary. It is not published. Do not upload it to PyPI
+until the maintainer chooses a public license and an account to publish from.

rowmonitor_airflow-0.1.0/PKG-INFO

@@ -0,0 +1,173 @@
+Metadata-Version: 2.4
+Name: rowmonitor-airflow
+Version: 0.1.0
+Summary: Beta Airflow listener that sends DAG run metadata to Row Monitor.
+Author: Row Monitor
+License-Expression: LicenseRef-RowMonitor-Proprietary
+Project-URL: Homepage, https://rowmonitor.com/help/airflow
+Project-URL: Repository, https://github.com/kharipara/row-monitor
+Project-URL: Documentation, https://rowmonitor.com/help/airflow
+Keywords: airflow,rowmonitor,monitoring
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Framework :: Apache Airflow
+Classifier: Topic :: System :: Monitoring
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: dev
+Requires-Dist: pytest>=8; extra == "dev"
+Dynamic: license-file
+
+# rowmonitor-airflow
+
+Beta listener for Airflow 2.6+ and 3.x. It sends DAG run metadata and a heartbeat of unpaused DAG schedules to [Row Monitor](https://rowmonitor.com/help/airflow).
+
+Version `0.1.0`. Not published yet. Not a general-availability release.
+
+Late runs are flagged within about 15 minutes. Row Monitor's check runs every 15 minutes. A run is late when it has not started within 5 minutes of the expected time.
+
+## What is sent
+
+Metadata only:
+
+- DAG id, run id, run type, logical date, start, end, state, duration
+- On task failure: DAG id, run id, task id, try number, start, end. Not the exception text
+- Heartbeat: unpaused DAG id, schedule, next run time when the scheduler has stored one
+
+Not sent: connections, variables, XComs, logs, DAG source, or row data.
+
+The HTTP call runs on a daemon thread with a short timeout (default 2 seconds). A failure does not fail the Airflow task.
+
+## Why the heartbeat is a thread
+
+The thread starts when the scheduler job starts (`on_starting`, and only in the scheduler process). It posts about once a minute.
+
+A small heartbeat DAG would wait for a free worker slot, and it would go silent if that DAG were paused, while every other DAG kept running. The thread still sends when slots are full, and it stops when the scheduler process stops. Row Monitor then opens one "heartbeat missing" issue after 15 minutes of silence, and it does not open a late or never-arrived issue per DAG for that outage.
+
+## Install
+
+The GitHub repository may be private. A plain git URL then fails. Do not commit a token.
+
+From a checkout of this repo (works for a private checkout you already have):
+
+```bash
+pip install plugins/rowmonitor-airflow
+```
+
+From git, when the repo is reachable:
+
+```bash
+pip install "git+https://github.com/kharipara/row-monitor.git#subdirectory=plugins/rowmonitor-airflow"
+```
+
+Private git, token only in the environment:
+
+```bash
+pip install "git+https://${GITHUB_TOKEN}@github.com/kharipara/row-monitor.git#subdirectory=plugins/rowmonitor-airflow"
+```
+
+Wheel, built from the checkout. CI also uploads this wheel as an artifact. It is not published.
+
+```bash
+python -m pip install build
+python -m build plugins/rowmonitor-airflow
+pip install plugins/rowmonitor-airflow/dist/rowmonitor_airflow-0.1.0-py3-none-any.whl
+```
+
+Install into the scheduler environment and every worker environment, then restart those processes.
+
+Airflow 2.6 through 2.10 and Airflow 3.x load `airflow.plugins`. Task-failure hook arguments differ (2.10 has `error` and `session`, earlier 2.x has `session`, Airflow 3 drops `session`). The listener accepts those shapes.
+
+## Environment
+
+| Variable | Required | Default | Purpose |
+| --- | --- | --- | --- |
+| `ROWMONITOR_URL` | yes | | Row Monitor origin, no path. Use the preview URL while testing. |
+| `ROWMONITOR_TOKEN` | yes | | Ingest token shown once when the deployment is created. |
+| `ROWMONITOR_HEARTBEAT` | no | `scheduler` | `scheduler`, `always`, or `off`. |
+| `ROWMONITOR_HEARTBEAT_SECONDS` | no | `60` | Minimum 30. |
+| `ROWMONITOR_TIMEOUT_SECONDS` | no | `2` | Capped at 5. |
+
+In the app: Airflow → create a deployment → copy the token into `ROWMONITOR_TOKEN`.
+
+## Local Docker test
+
+One compose file, `examples/docker/docker-compose.yml`. It is a sample, not a production Airflow. Run steps, the 3.x image, and cleanup are in `examples/docker/README.md`.
+
+```bash
+cd plugins/rowmonitor-airflow/examples/docker
+cp .env.example .env
+# Edit ROWMONITOR_TOKEN. Never commit the real token.
+# .env.example sets ROWMONITOR_URL=https://rowmonitor.com
+
+# Airflow 2.10.5. UI on http://localhost:8080 when standalone is ready.
+docker compose up
+
+# Airflow 3.4.0. Same port. airflow standalone also starts the dag-processor and api-server.
+AIRFLOW_IMAGE=apache/airflow:3.4.0 docker compose up
+```
+
+Airflow 2.10.5 was tested by a user end to end against https://rowmonitor.com with this compose file. All checks passed. Airflow 3.4.0 is covered by the plugin unit tests. It was not part of that end-to-end run. Run one image at a time, and `docker compose down -v` before switching so the database is empty.
+
+The container copies the read-only checkout to `/tmp` and runs `pip install` there, then `airflow standalone`. The admin user is `admin`. The password is printed in the service logs. Sample DAGs are mounted from `examples/dags/`:
+
+| DAG | What it does | What to expect after the check |
+| --- | --- | --- |
+| `rowmonitor_failed` | Raises every 2 minutes | FAILED |
+| `rowmonitor_late` | `max_active_runs=1`, sleeps 8 minutes, catchup on for a few minutes | LATE once the blocked run starts (start is more than 5 minutes after the logical date) |
+| `rowmonitor_slow` | Five 20-second runs, then a 15-minute run | SLOW after the long run succeeds. Needs the five successes first |
+| `rowmonitor_never_arrived` | `catchup=False`, one run sleeps 40 minutes, schedule every minute | NEVER_ARRIVED for minutes the scheduler did not create. The heartbeat keeps listing the DAG |
+
+Pausing a DAG removes it from the unpaused heartbeat. That is not the never-arrived test.
+
+Heartbeat missing: from `examples/docker`, `docker compose stop airflow` for at least 15 minutes. Leave the deployment token in place, then run the check. You should see one heartbeat-missing issue, not a late issue for every DAG. Cleanup is `docker compose down -v`.
+
+### How long to wait
+
+The production cron is every 15 minutes and starts only after this change is merged. Preview deployments do not run that schedule by themselves.
+
+After the DAG has done the thing you care about, trigger the check if `CRON_SECRET` is set on the preview:
+
+```bash
+curl -sS -X POST \
+  -H "Authorization: Bearer $CRON_SECRET" \
+  "https://YOUR-PREVIEW.vercel.app/api/cron/airflow"
+```
+
+GET works the same way. Do not print the secret. If the preview has no `CRON_SECRET`, the route returns 401 and the check has to wait until production cron, or be run against a local app that has the secret.
+
+Rough waits before that manual check is worth calling:
+
+- Failed: the first `rowmonitor_failed` run, usually under 2 minutes, then the check
+- Late: about 8 minutes for the sleep, then the check
+- Never arrived: about 6 minutes after the blocker starts (the 5-minute grace), then the check
+- Slow: about 6 short minutes plus the 15-minute sleep, then the check
+- Heartbeat missing: 15 minutes with the scheduler stopped, then the check
+
+In the app, open Airflow. You should see the deployment's DAGs, last run, state, duration, next expected time, and open issues. Those times, and the alert emails and Slack messages, use the workspace time zone. It starts as the browser's zone at signup. The owner can change it in Settings, and daylight saving is included. Airflow schedules are still evaluated the way Airflow defines them. The time zone does not change detection. Over the Free limit (5 DAGs) the extra DAGs stay listed as over the limit and are not checked.
+
+## Tests
+
+```bash
+cd plugins/rowmonitor-airflow
+python -m pip install pytest
+python -m pip install -e .
+python -m pytest -q
+```
+
+`tests/test_airflow_install.py` runs only when `apache-airflow` is installed in that interpreter. Unit tests do not need Airflow.
+
+## Publishing to PyPI later
+
+Do not publish from this change. After merge, publish with Actions, not an API token.
+
+`.github/workflows/publish-airflow-plugin.yml` is `workflow_dispatch` only. It builds the sdist and wheel, runs `twine check`, and the `publish` job uses `pypa/gh-action-pypi-publish` with OIDC against the `pypi` environment. That workflow name appears in the Actions tab only after the file is on `main`.
+
+Until that upload, install from git or from a wheel. Then `pip install rowmonitor-airflow`.

rowmonitor_airflow-0.1.0/README.md

@@ -0,0 +1,146 @@
+# rowmonitor-airflow
+
+Beta listener for Airflow 2.6+ and 3.x. It sends DAG run metadata and a heartbeat of unpaused DAG schedules to [Row Monitor](https://rowmonitor.com/help/airflow).
+
+Version `0.1.0`. Not published yet. Not a general-availability release.
+
+Late runs are flagged within about 15 minutes. Row Monitor's check runs every 15 minutes. A run is late when it has not started within 5 minutes of the expected time.
+
+## What is sent
+
+Metadata only:
+
+- DAG id, run id, run type, logical date, start, end, state, duration
+- On task failure: DAG id, run id, task id, try number, start, end. Not the exception text
+- Heartbeat: unpaused DAG id, schedule, next run time when the scheduler has stored one
+
+Not sent: connections, variables, XComs, logs, DAG source, or row data.
+
+The HTTP call runs on a daemon thread with a short timeout (default 2 seconds). A failure does not fail the Airflow task.
+
+## Why the heartbeat is a thread
+
+The thread starts when the scheduler job starts (`on_starting`, and only in the scheduler process). It posts about once a minute.
+
+A small heartbeat DAG would wait for a free worker slot, and it would go silent if that DAG were paused, while every other DAG kept running. The thread still sends when slots are full, and it stops when the scheduler process stops. Row Monitor then opens one "heartbeat missing" issue after 15 minutes of silence, and it does not open a late or never-arrived issue per DAG for that outage.
+
+## Install
+
+The GitHub repository may be private. A plain git URL then fails. Do not commit a token.
+
+From a checkout of this repo (works for a private checkout you already have):
+
+```bash
+pip install plugins/rowmonitor-airflow
+```
+
+From git, when the repo is reachable:
+
+```bash
+pip install "git+https://github.com/kharipara/row-monitor.git#subdirectory=plugins/rowmonitor-airflow"
+```
+
+Private git, token only in the environment:
+
+```bash
+pip install "git+https://${GITHUB_TOKEN}@github.com/kharipara/row-monitor.git#subdirectory=plugins/rowmonitor-airflow"
+```
+
+Wheel, built from the checkout. CI also uploads this wheel as an artifact. It is not published.
+
+```bash
+python -m pip install build
+python -m build plugins/rowmonitor-airflow
+pip install plugins/rowmonitor-airflow/dist/rowmonitor_airflow-0.1.0-py3-none-any.whl
+```
+
+Install into the scheduler environment and every worker environment, then restart those processes.
+
+Airflow 2.6 through 2.10 and Airflow 3.x load `airflow.plugins`. Task-failure hook arguments differ (2.10 has `error` and `session`, earlier 2.x has `session`, Airflow 3 drops `session`). The listener accepts those shapes.
+
+## Environment
+
+| Variable | Required | Default | Purpose |
+| --- | --- | --- | --- |
+| `ROWMONITOR_URL` | yes | | Row Monitor origin, no path. Use the preview URL while testing. |
+| `ROWMONITOR_TOKEN` | yes | | Ingest token shown once when the deployment is created. |
+| `ROWMONITOR_HEARTBEAT` | no | `scheduler` | `scheduler`, `always`, or `off`. |
+| `ROWMONITOR_HEARTBEAT_SECONDS` | no | `60` | Minimum 30. |
+| `ROWMONITOR_TIMEOUT_SECONDS` | no | `2` | Capped at 5. |
+
+In the app: Airflow → create a deployment → copy the token into `ROWMONITOR_TOKEN`.
+
+## Local Docker test
+
+One compose file, `examples/docker/docker-compose.yml`. It is a sample, not a production Airflow. Run steps, the 3.x image, and cleanup are in `examples/docker/README.md`.
+
+```bash
+cd plugins/rowmonitor-airflow/examples/docker
+cp .env.example .env
+# Edit ROWMONITOR_TOKEN. Never commit the real token.
+# .env.example sets ROWMONITOR_URL=https://rowmonitor.com
+
+# Airflow 2.10.5. UI on http://localhost:8080 when standalone is ready.
+docker compose up
+
+# Airflow 3.4.0. Same port. airflow standalone also starts the dag-processor and api-server.
+AIRFLOW_IMAGE=apache/airflow:3.4.0 docker compose up
+```
+
+Airflow 2.10.5 was tested by a user end to end against https://rowmonitor.com with this compose file. All checks passed. Airflow 3.4.0 is covered by the plugin unit tests. It was not part of that end-to-end run. Run one image at a time, and `docker compose down -v` before switching so the database is empty.
+
+The container copies the read-only checkout to `/tmp` and runs `pip install` there, then `airflow standalone`. The admin user is `admin`. The password is printed in the service logs. Sample DAGs are mounted from `examples/dags/`:
+
+| DAG | What it does | What to expect after the check |
+| --- | --- | --- |
+| `rowmonitor_failed` | Raises every 2 minutes | FAILED |
+| `rowmonitor_late` | `max_active_runs=1`, sleeps 8 minutes, catchup on for a few minutes | LATE once the blocked run starts (start is more than 5 minutes after the logical date) |
+| `rowmonitor_slow` | Five 20-second runs, then a 15-minute run | SLOW after the long run succeeds. Needs the five successes first |
+| `rowmonitor_never_arrived` | `catchup=False`, one run sleeps 40 minutes, schedule every minute | NEVER_ARRIVED for minutes the scheduler did not create. The heartbeat keeps listing the DAG |
+
+Pausing a DAG removes it from the unpaused heartbeat. That is not the never-arrived test.
+
+Heartbeat missing: from `examples/docker`, `docker compose stop airflow` for at least 15 minutes. Leave the deployment token in place, then run the check. You should see one heartbeat-missing issue, not a late issue for every DAG. Cleanup is `docker compose down -v`.
+
+### How long to wait
+
+The production cron is every 15 minutes and starts only after this change is merged. Preview deployments do not run that schedule by themselves.
+
+After the DAG has done the thing you care about, trigger the check if `CRON_SECRET` is set on the preview:
+
+```bash
+curl -sS -X POST \
+  -H "Authorization: Bearer $CRON_SECRET" \
+  "https://YOUR-PREVIEW.vercel.app/api/cron/airflow"
+```
+
+GET works the same way. Do not print the secret. If the preview has no `CRON_SECRET`, the route returns 401 and the check has to wait until production cron, or be run against a local app that has the secret.
+
+Rough waits before that manual check is worth calling:
+
+- Failed: the first `rowmonitor_failed` run, usually under 2 minutes, then the check
+- Late: about 8 minutes for the sleep, then the check
+- Never arrived: about 6 minutes after the blocker starts (the 5-minute grace), then the check
+- Slow: about 6 short minutes plus the 15-minute sleep, then the check
+- Heartbeat missing: 15 minutes with the scheduler stopped, then the check
+
+In the app, open Airflow. You should see the deployment's DAGs, last run, state, duration, next expected time, and open issues. Those times, and the alert emails and Slack messages, use the workspace time zone. It starts as the browser's zone at signup. The owner can change it in Settings, and daylight saving is included. Airflow schedules are still evaluated the way Airflow defines them. The time zone does not change detection. Over the Free limit (5 DAGs) the extra DAGs stay listed as over the limit and are not checked.
+
+## Tests
+
+```bash
+cd plugins/rowmonitor-airflow
+python -m pip install pytest
+python -m pip install -e .
+python -m pytest -q
+```
+
+`tests/test_airflow_install.py` runs only when `apache-airflow` is installed in that interpreter. Unit tests do not need Airflow.
+
+## Publishing to PyPI later
+
+Do not publish from this change. After merge, publish with Actions, not an API token.
+
+`.github/workflows/publish-airflow-plugin.yml` is `workflow_dispatch` only. It builds the sdist and wheel, runs `twine check`, and the `publish` job uses `pypa/gh-action-pypi-publish` with OIDC against the `pypi` environment. That workflow name appears in the Actions tab only after the file is on `main`.
+
+Until that upload, install from git or from a wheel. Then `pip install rowmonitor-airflow`.

rowmonitor_airflow-0.1.0/pyproject.toml

@@ -0,0 +1,43 @@
+[project]
+name = "rowmonitor-airflow"
+version = "0.1.0"
+description = "Beta Airflow listener that sends DAG run metadata to Row Monitor."
+readme = "README.md"
+license = "LicenseRef-RowMonitor-Proprietary"
+requires-python = ">=3.9"
+authors = [{ name = "Row Monitor" }]
+keywords = ["airflow", "rowmonitor", "monitoring"]
+classifiers = [
+  "Development Status :: 4 - Beta",
+  "Intended Audience :: Developers",
+  "Programming Language :: Python :: 3",
+  "Programming Language :: Python :: 3.9",
+  "Programming Language :: Python :: 3.10",
+  "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: 3.12",
+  "Programming Language :: Python :: 3.13",
+  "Framework :: Apache Airflow",
+  "Topic :: System :: Monitoring",
+]
+dependencies = []
+
+[project.urls]
+Homepage = "https://rowmonitor.com/help/airflow"
+Repository = "https://github.com/kharipara/row-monitor"
+Documentation = "https://rowmonitor.com/help/airflow"
+
+[project.entry-points."airflow.plugins"]
+rowmonitor_airflow = "rowmonitor_airflow.plugin:RowMonitorPlugin"
+
+[project.optional-dependencies]
+dev = ["pytest>=8"]
+
+[build-system]
+requires = ["setuptools>=68"]
+build-backend = "setuptools.build_meta"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]

rowmonitor_airflow-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

rowmonitor_airflow-0.1.0/src/rowmonitor_airflow/__init__.py

@@ -0,0 +1,3 @@
+"""Row Monitor listener for Airflow 2.6+ and 3.x. Beta. Metadata only."""
+
+__version__ = "0.1.0"

rowmonitor_airflow-0.1.0/src/rowmonitor_airflow/client.py

@@ -0,0 +1,60 @@
+"""Non-blocking HTTP. Failures are swallowed. The token is never logged."""
+
+from __future__ import annotations
+
+import json
+import os
+import threading
+import urllib.error
+import urllib.request
+
+_warned = False
+
+
+def load_config():
+    url = os.environ.get("ROWMONITOR_URL", "").strip().rstrip("/")
+    token = os.environ.get("ROWMONITOR_TOKEN", "").strip()
+    try:
+        timeout = float(os.environ.get("ROWMONITOR_TIMEOUT_SECONDS", "2"))
+    except ValueError:
+        timeout = 2.0
+    timeout = min(max(timeout, 0.2), 5.0)
+    return url, token, timeout
+
+
+def post_json(path, payload, opener=None):
+    global _warned
+    url, token, timeout = load_config()
+    if not url or not token:
+        if not _warned:
+            _warned = True
+        return
+    body = json.dumps(payload).encode("utf-8")
+    if len(body) > 250_000:
+        return
+    request = urllib.request.Request(
+        url + path,
+        data=body,
+        headers={
+            "Authorization": "Bearer " + token,
+            "Content-Type": "application/json",
+            "User-Agent": "rowmonitor-airflow/0.1.0",
+        },
+        method="POST",
+    )
+    open_url = opener or urllib.request.urlopen
+    try:
+        with open_url(request, timeout=timeout) as response:
+            response.read(2048)
+    except Exception:
+        return
+
+
+def emit_async(path, payload, sender=post_json):
+    def _safe():
+        try:
+            sender(path, payload)
+        except Exception:
+            return
+
+    threading.Thread(target=_safe, name="rowmonitor-emit", daemon=True).start()

rowmonitor_airflow-0.1.0/src/rowmonitor_airflow/collect.py

@@ -0,0 +1,36 @@
+"""Read unpaused DAG schedules from the Airflow metadata database."""
+
+from __future__ import annotations
+
+
+def query_unpaused():
+    from airflow.models.dag import DagModel
+    from airflow.utils.session import create_session
+
+    rows = []
+    with create_session() as session:
+        query = session.query(DagModel)
+        if hasattr(DagModel, "is_active"):
+            query = query.filter(DagModel.is_active.is_(True))
+        if hasattr(DagModel, "is_paused"):
+            query = query.filter(DagModel.is_paused.is_(False))
+        for row in query.all():
+            rows.append(
+                SimpleDag(
+                    dag_id=row.dag_id,
+                    is_paused=bool(getattr(row, "is_paused", False)),
+                    timetable_description=getattr(row, "timetable_description", None),
+                    schedule_interval=getattr(row, "schedule_interval", None),
+                    next_dagrun=getattr(row, "next_dagrun", None),
+                )
+            )
+    return rows
+
+
+class SimpleDag:
+    def __init__(self, dag_id, is_paused, timetable_description, schedule_interval, next_dagrun):
+        self.dag_id = dag_id
+        self.is_paused = is_paused
+        self.timetable_description = timetable_description
+        self.schedule_interval = schedule_interval
+        self.next_dagrun = next_dagrun

rowmonitor_airflow-0.1.0/src/rowmonitor_airflow/heartbeat.py

@@ -0,0 +1,84 @@
+"""Scheduler heartbeat.
+
+A daemon thread is started from the scheduler job, not a DAG.
+
+A heartbeat DAG would wait for a worker slot and would go silent if that one
+DAG were paused, while the rest of the deployment kept running. The thread
+sends even when task slots are full, and it dies with the scheduler process.
+That is the outage Row Monitor reports as one missing heartbeat. Duplicate
+schedulers send duplicate heartbeats. The ingest API treats that as a state
+update, not a second deployment.
+"""
+
+from __future__ import annotations
+
+import os
+import threading
+
+from rowmonitor_airflow.client import emit_async
+from rowmonitor_airflow.metadata import heartbeat_body
+
+_lock = threading.Lock()
+_started = False
+_stop = threading.Event()
+
+
+def heartbeat_interval():
+    try:
+        seconds = float(os.environ.get("ROWMONITOR_HEARTBEAT_SECONDS", "60"))
+    except ValueError:
+        seconds = 60.0
+    return max(30.0, seconds)
+
+
+def heartbeat_enabled():
+    mode = os.environ.get("ROWMONITOR_HEARTBEAT", "scheduler").strip().lower()
+    return mode not in {"0", "off", "false", "no"}
+
+
+def collect_unpaused():
+    from rowmonitor_airflow.collect import query_unpaused
+
+    return query_unpaused()
+
+
+def heartbeat_loop(stop_event, collect, send, interval):
+    while not stop_event.is_set():
+        try:
+            rows = collect()
+            body = heartbeat_body(rows)
+            send("/api/airflow/v1/heartbeat", body)
+        except Exception:
+            pass
+        if stop_event.wait(interval):
+            return
+
+
+def start_heartbeat(collect=None, send=None):
+    global _started
+    if not heartbeat_enabled():
+        return False
+    with _lock:
+        if _started:
+            return False
+        _started = True
+    threading.Thread(
+        target=heartbeat_loop,
+        kwargs={
+            "stop_event": _stop,
+            "collect": collect or collect_unpaused,
+            "send": send or emit_async,
+            "interval": heartbeat_interval(),
+        },
+        name="rowmonitor-heartbeat",
+        daemon=True,
+    ).start()
+    return True
+
+
+def reset_heartbeat_for_tests():
+    global _started
+    _stop.set()
+    with _lock:
+        _started = False
+    _stop.clear()

rowmonitor_airflow-0.1.0/src/rowmonitor_airflow/listener.py

@@ -0,0 +1,72 @@
+"""Listener hooks for Airflow 2.6+ and 3.x.
+
+Airflow 2.10 task-failure hooks take (previous_state, task_instance, error, session).
+Earlier 2.x hooks take (previous_state, task_instance, session) and have no error.
+Airflow 3 dropped the session argument and kept error. **kwargs covers that.
+The exception text is not sent. It can contain query text or row values.
+"""
+
+from __future__ import annotations
+
+import sys
+
+from rowmonitor_airflow.client import emit_async
+from rowmonitor_airflow.heartbeat import start_heartbeat
+from rowmonitor_airflow.metadata import dag_run_payload, task_failure_payload
+
+
+def _mark(func):
+    try:
+        from airflow.listeners import hookimpl
+    except ImportError:
+        return func
+    return hookimpl(func)
+
+
+def _guard(fn):
+    try:
+        fn()
+    except Exception:
+        return
+
+
+def _argv_is_scheduler():
+    command = " ".join(sys.argv).lower()
+    return "scheduler" in command
+
+
+def _on_dag_run_running(dag_run, msg="", **kwargs):
+    _guard(lambda: emit_async("/api/airflow/v1/events", dag_run_payload(dag_run, "running")))
+
+
+def _on_dag_run_success(dag_run, msg="", **kwargs):
+    _guard(lambda: emit_async("/api/airflow/v1/events", dag_run_payload(dag_run, "success")))
+
+
+def _on_dag_run_failed(dag_run, msg="", **kwargs):
+    _guard(lambda: emit_async("/api/airflow/v1/events", dag_run_payload(dag_run, "failed")))
+
+
+def _on_task_instance_failed(previous_state, task_instance, **kwargs):
+    def _send():
+        payload = task_failure_payload(task_instance)
+        if payload is not None:
+            emit_async("/api/airflow/v1/events", payload)
+
+    _guard(_send)
+
+
+def _on_starting(component=None, **kwargs):
+    try:
+        name = type(component).__name__.lower() if component is not None else ""
+        if "scheduler" in name or _argv_is_scheduler():
+            start_heartbeat()
+    except Exception:
+        return
+
+
+on_dag_run_running = _mark(_on_dag_run_running)
+on_dag_run_success = _mark(_on_dag_run_success)
+on_dag_run_failed = _mark(_on_dag_run_failed)
+on_task_instance_failed = _mark(_on_task_instance_failed)
+on_starting = _mark(_on_starting)

rowmonitor_airflow-0.1.0/src/rowmonitor_airflow/metadata.py

@@ -0,0 +1,131 @@
+"""Build ingest payloads. The allowlist is the whole contract."""
+
+from __future__ import annotations
+
+from datetime import datetime, timezone
+
+DAG_RUN_KEYS = ("type", "dag_id", "run_id", "run_type", "logical_date", "state", "start_date", "end_date", "duration_ms")
+TASK_KEYS = ("type", "dag_id", "run_id", "task_id", "state", "start_date", "end_date", "try_number")
+HEARTBEAT_KEYS = ("dag_id", "schedule", "next_run", "paused")
+
+
+def iso(value):
+    if value is None:
+        return None
+    if isinstance(value, datetime):
+        if value.tzinfo is None:
+            value = value.replace(tzinfo=timezone.utc)
+        return value.isoformat()
+    isoformat = getattr(value, "isoformat", None)
+    if callable(isoformat):
+        try:
+            text = isoformat()
+        except Exception:
+            return None
+        if isinstance(text, str) and text and "T" in text:
+            if text.endswith("Z") or "+" in text[10:] or "-" in text[10:]:
+                return text
+            return text + "Z" if "+" not in text else text
+    return None
+
+
+def duration_ms(start, end):
+    if start is None or end is None:
+        return None
+    try:
+        seconds = (end - start).total_seconds()
+    except Exception:
+        return None
+    if seconds < 0:
+        return None
+    return int(seconds * 1000)
+
+
+def _state(value, fallback):
+    raw = getattr(value, "value", value)
+    text = str(raw or fallback).lower()
+    if "." in text:
+        text = text.split(".")[-1]
+    return text
+
+
+def dag_run_payload(dag_run, state):
+    logical = getattr(dag_run, "logical_date", None)
+    if logical is None:
+        logical = getattr(dag_run, "execution_date", None)
+    start = getattr(dag_run, "start_date", None)
+    end = getattr(dag_run, "end_date", None)
+    run_type = getattr(dag_run, "run_type", None)
+    if run_type is not None:
+        run_type = str(getattr(run_type, "value", run_type))[:64]
+    payload = {
+        "type": "dag_run",
+        "dag_id": str(getattr(dag_run, "dag_id", ""))[:250],
+        "run_id": str(getattr(dag_run, "run_id", ""))[:250],
+        "run_type": run_type,
+        "logical_date": iso(logical),
+        "state": _state(state, "running"),
+        "start_date": iso(start),
+        "end_date": iso(end),
+        "duration_ms": duration_ms(start, end),
+    }
+    return {key: payload[key] for key in DAG_RUN_KEYS}
+
+
+def task_failure_payload(task_instance):
+    dag_id = getattr(task_instance, "dag_id", None)
+    task_id = getattr(task_instance, "task_id", None)
+    run_id = getattr(task_instance, "run_id", None)
+    if not run_id:
+        dag_run = getattr(task_instance, "dag_run", None)
+        run_id = getattr(dag_run, "run_id", None) if dag_run is not None else None
+    if not dag_id or not task_id or not run_id:
+        return None
+    try_number = getattr(task_instance, "try_number", None)
+    if try_number is not None:
+        try:
+            try_number = int(try_number)
+        except (TypeError, ValueError):
+            try_number = None
+    payload = {
+        "type": "task_run",
+        "dag_id": str(dag_id)[:250],
+        "run_id": str(run_id)[:250],
+        "task_id": str(task_id)[:250],
+        "state": "failed",
+        "start_date": iso(getattr(task_instance, "start_date", None)),
+        "end_date": iso(getattr(task_instance, "end_date", None)),
+        "try_number": try_number,
+    }
+    return {key: payload[key] for key in TASK_KEYS}
+
+
+def snapshot_dag(row):
+    dag_id = getattr(row, "dag_id", None)
+    if not dag_id or getattr(row, "is_paused", False):
+        return None
+    schedule = None
+    for attr in ("timetable_description", "schedule_interval", "timetable_summary"):
+        value = getattr(row, attr, None)
+        if value is None:
+            continue
+        text = str(value).strip()
+        if text and text.lower() not in {"none", "null"}:
+            schedule = text[:500]
+            break
+    return {
+        "dag_id": str(dag_id)[:250],
+        "schedule": schedule,
+        "next_run": iso(getattr(row, "next_dagrun", None)),
+        "paused": False,
+    }
+
+
+def heartbeat_body(rows, limit=500):
+    dags = []
+    for row in rows:
+        snap = snapshot_dag(row)
+        if snap is not None:
+            dags.append({key: snap[key] for key in HEARTBEAT_KEYS})
+    truncated = len(dags) > limit
+    return {"dags": dags[:limit], "truncated": truncated}

rowmonitor_airflow-0.1.0/src/rowmonitor_airflow/plugin.py

@@ -0,0 +1,21 @@
+"""Airflow plugin entry point. Importing this module requires Airflow."""
+
+from __future__ import annotations
+
+
+def _load():
+    from airflow.plugins_manager import AirflowPlugin
+
+    from rowmonitor_airflow import listener
+
+    class RowMonitorPlugin(AirflowPlugin):
+        name = "rowmonitor_airflow"
+        listeners = [listener]
+
+    return RowMonitorPlugin
+
+
+try:
+    RowMonitorPlugin = _load()
+except ImportError:
+    RowMonitorPlugin = None  # type: ignore[misc, assignment]

rowmonitor_airflow-0.1.0/src/rowmonitor_airflow.egg-info/PKG-INFO

@@ -0,0 +1,173 @@
+Metadata-Version: 2.4
+Name: rowmonitor-airflow
+Version: 0.1.0
+Summary: Beta Airflow listener that sends DAG run metadata to Row Monitor.
+Author: Row Monitor
+License-Expression: LicenseRef-RowMonitor-Proprietary
+Project-URL: Homepage, https://rowmonitor.com/help/airflow
+Project-URL: Repository, https://github.com/kharipara/row-monitor
+Project-URL: Documentation, https://rowmonitor.com/help/airflow
+Keywords: airflow,rowmonitor,monitoring
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Framework :: Apache Airflow
+Classifier: Topic :: System :: Monitoring
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: dev
+Requires-Dist: pytest>=8; extra == "dev"
+Dynamic: license-file
+
+# rowmonitor-airflow
+
+Beta listener for Airflow 2.6+ and 3.x. It sends DAG run metadata and a heartbeat of unpaused DAG schedules to [Row Monitor](https://rowmonitor.com/help/airflow).
+
+Version `0.1.0`. Not published yet. Not a general-availability release.
+
+Late runs are flagged within about 15 minutes. Row Monitor's check runs every 15 minutes. A run is late when it has not started within 5 minutes of the expected time.
+
+## What is sent
+
+Metadata only:
+
+- DAG id, run id, run type, logical date, start, end, state, duration
+- On task failure: DAG id, run id, task id, try number, start, end. Not the exception text
+- Heartbeat: unpaused DAG id, schedule, next run time when the scheduler has stored one
+
+Not sent: connections, variables, XComs, logs, DAG source, or row data.
+
+The HTTP call runs on a daemon thread with a short timeout (default 2 seconds). A failure does not fail the Airflow task.
+
+## Why the heartbeat is a thread
+
+The thread starts when the scheduler job starts (`on_starting`, and only in the scheduler process). It posts about once a minute.
+
+A small heartbeat DAG would wait for a free worker slot, and it would go silent if that DAG were paused, while every other DAG kept running. The thread still sends when slots are full, and it stops when the scheduler process stops. Row Monitor then opens one "heartbeat missing" issue after 15 minutes of silence, and it does not open a late or never-arrived issue per DAG for that outage.
+
+## Install
+
+The GitHub repository may be private. A plain git URL then fails. Do not commit a token.
+
+From a checkout of this repo (works for a private checkout you already have):
+
+```bash
+pip install plugins/rowmonitor-airflow
+```
+
+From git, when the repo is reachable:
+
+```bash
+pip install "git+https://github.com/kharipara/row-monitor.git#subdirectory=plugins/rowmonitor-airflow"
+```
+
+Private git, token only in the environment:
+
+```bash
+pip install "git+https://${GITHUB_TOKEN}@github.com/kharipara/row-monitor.git#subdirectory=plugins/rowmonitor-airflow"
+```
+
+Wheel, built from the checkout. CI also uploads this wheel as an artifact. It is not published.
+
+```bash
+python -m pip install build
+python -m build plugins/rowmonitor-airflow
+pip install plugins/rowmonitor-airflow/dist/rowmonitor_airflow-0.1.0-py3-none-any.whl
+```
+
+Install into the scheduler environment and every worker environment, then restart those processes.
+
+Airflow 2.6 through 2.10 and Airflow 3.x load `airflow.plugins`. Task-failure hook arguments differ (2.10 has `error` and `session`, earlier 2.x has `session`, Airflow 3 drops `session`). The listener accepts those shapes.
+
+## Environment
+
+| Variable | Required | Default | Purpose |
+| --- | --- | --- | --- |
+| `ROWMONITOR_URL` | yes | | Row Monitor origin, no path. Use the preview URL while testing. |
+| `ROWMONITOR_TOKEN` | yes | | Ingest token shown once when the deployment is created. |
+| `ROWMONITOR_HEARTBEAT` | no | `scheduler` | `scheduler`, `always`, or `off`. |
+| `ROWMONITOR_HEARTBEAT_SECONDS` | no | `60` | Minimum 30. |
+| `ROWMONITOR_TIMEOUT_SECONDS` | no | `2` | Capped at 5. |
+
+In the app: Airflow → create a deployment → copy the token into `ROWMONITOR_TOKEN`.
+
+## Local Docker test
+
+One compose file, `examples/docker/docker-compose.yml`. It is a sample, not a production Airflow. Run steps, the 3.x image, and cleanup are in `examples/docker/README.md`.
+
+```bash
+cd plugins/rowmonitor-airflow/examples/docker
+cp .env.example .env
+# Edit ROWMONITOR_TOKEN. Never commit the real token.
+# .env.example sets ROWMONITOR_URL=https://rowmonitor.com
+
+# Airflow 2.10.5. UI on http://localhost:8080 when standalone is ready.
+docker compose up
+
+# Airflow 3.4.0. Same port. airflow standalone also starts the dag-processor and api-server.
+AIRFLOW_IMAGE=apache/airflow:3.4.0 docker compose up
+```
+
+Airflow 2.10.5 was tested by a user end to end against https://rowmonitor.com with this compose file. All checks passed. Airflow 3.4.0 is covered by the plugin unit tests. It was not part of that end-to-end run. Run one image at a time, and `docker compose down -v` before switching so the database is empty.
+
+The container copies the read-only checkout to `/tmp` and runs `pip install` there, then `airflow standalone`. The admin user is `admin`. The password is printed in the service logs. Sample DAGs are mounted from `examples/dags/`:
+
+| DAG | What it does | What to expect after the check |
+| --- | --- | --- |
+| `rowmonitor_failed` | Raises every 2 minutes | FAILED |
+| `rowmonitor_late` | `max_active_runs=1`, sleeps 8 minutes, catchup on for a few minutes | LATE once the blocked run starts (start is more than 5 minutes after the logical date) |
+| `rowmonitor_slow` | Five 20-second runs, then a 15-minute run | SLOW after the long run succeeds. Needs the five successes first |
+| `rowmonitor_never_arrived` | `catchup=False`, one run sleeps 40 minutes, schedule every minute | NEVER_ARRIVED for minutes the scheduler did not create. The heartbeat keeps listing the DAG |
+
+Pausing a DAG removes it from the unpaused heartbeat. That is not the never-arrived test.
+
+Heartbeat missing: from `examples/docker`, `docker compose stop airflow` for at least 15 minutes. Leave the deployment token in place, then run the check. You should see one heartbeat-missing issue, not a late issue for every DAG. Cleanup is `docker compose down -v`.
+
+### How long to wait
+
+The production cron is every 15 minutes and starts only after this change is merged. Preview deployments do not run that schedule by themselves.
+
+After the DAG has done the thing you care about, trigger the check if `CRON_SECRET` is set on the preview:
+
+```bash
+curl -sS -X POST \
+  -H "Authorization: Bearer $CRON_SECRET" \
+  "https://YOUR-PREVIEW.vercel.app/api/cron/airflow"
+```
+
+GET works the same way. Do not print the secret. If the preview has no `CRON_SECRET`, the route returns 401 and the check has to wait until production cron, or be run against a local app that has the secret.
+
+Rough waits before that manual check is worth calling:
+
+- Failed: the first `rowmonitor_failed` run, usually under 2 minutes, then the check
+- Late: about 8 minutes for the sleep, then the check
+- Never arrived: about 6 minutes after the blocker starts (the 5-minute grace), then the check
+- Slow: about 6 short minutes plus the 15-minute sleep, then the check
+- Heartbeat missing: 15 minutes with the scheduler stopped, then the check
+
+In the app, open Airflow. You should see the deployment's DAGs, last run, state, duration, next expected time, and open issues. Those times, and the alert emails and Slack messages, use the workspace time zone. It starts as the browser's zone at signup. The owner can change it in Settings, and daylight saving is included. Airflow schedules are still evaluated the way Airflow defines them. The time zone does not change detection. Over the Free limit (5 DAGs) the extra DAGs stay listed as over the limit and are not checked.
+
+## Tests
+
+```bash
+cd plugins/rowmonitor-airflow
+python -m pip install pytest
+python -m pip install -e .
+python -m pytest -q
+```
+
+`tests/test_airflow_install.py` runs only when `apache-airflow` is installed in that interpreter. Unit tests do not need Airflow.
+
+## Publishing to PyPI later
+
+Do not publish from this change. After merge, publish with Actions, not an API token.
+
+`.github/workflows/publish-airflow-plugin.yml` is `workflow_dispatch` only. It builds the sdist and wheel, runs `twine check`, and the `publish` job uses `pypa/gh-action-pypi-publish` with OIDC against the `pypi` environment. That workflow name appears in the Actions tab only after the file is on `main`.
+
+Until that upload, install from git or from a wheel. Then `pip install rowmonitor-airflow`.

rowmonitor_airflow-0.1.0/src/rowmonitor_airflow.egg-info/SOURCES.txt

@@ -0,0 +1,20 @@
+LICENSE
+README.md
+pyproject.toml
+src/rowmonitor_airflow/__init__.py
+src/rowmonitor_airflow/client.py
+src/rowmonitor_airflow/collect.py
+src/rowmonitor_airflow/heartbeat.py
+src/rowmonitor_airflow/listener.py
+src/rowmonitor_airflow/metadata.py
+src/rowmonitor_airflow/plugin.py
+src/rowmonitor_airflow.egg-info/PKG-INFO
+src/rowmonitor_airflow.egg-info/SOURCES.txt
+src/rowmonitor_airflow.egg-info/dependency_links.txt
+src/rowmonitor_airflow.egg-info/entry_points.txt
+src/rowmonitor_airflow.egg-info/requires.txt
+src/rowmonitor_airflow.egg-info/top_level.txt
+tests/test_airflow_install.py
+tests/test_client.py
+tests/test_listener.py
+tests/test_metadata.py

rowmonitor_airflow-0.1.0/src/rowmonitor_airflow.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

rowmonitor_airflow-0.1.0/src/rowmonitor_airflow.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[airflow.plugins]
+rowmonitor_airflow = rowmonitor_airflow.plugin:RowMonitorPlugin

rowmonitor_airflow-0.1.0/src/rowmonitor_airflow.egg-info/requires.txt

@@ -0,0 +1,3 @@
+
+[dev]
+pytest>=8

rowmonitor_airflow-0.1.0/src/rowmonitor_airflow.egg-info/top_level.txt

@@ -0,0 +1 @@
+rowmonitor_airflow

rowmonitor_airflow-0.1.0/tests/test_airflow_install.py

@@ -0,0 +1,31 @@
+"""Runs only when apache-airflow is installed in this interpreter."""
+
+import pytest
+
+pytest.importorskip("airflow")
+
+
+def test_listener_registers_with_this_airflow():
+    import airflow
+    from airflow.plugins_manager import AirflowPlugin
+
+    from rowmonitor_airflow.plugin import RowMonitorPlugin
+
+    assert issubclass(RowMonitorPlugin, AirflowPlugin)
+    assert RowMonitorPlugin.listeners
+    hooks = [
+        "on_dag_run_running",
+        "on_dag_run_success",
+        "on_dag_run_failed",
+        "on_task_instance_failed",
+        "on_starting",
+    ]
+    listener = RowMonitorPlugin.listeners[0]
+    for name in hooks:
+        assert callable(getattr(listener, name))
+    # Importing the listener manager proves the hook specs load on this version.
+    from airflow.listeners.listener import get_listener_manager
+
+    manager = get_listener_manager()
+    manager.add_listener(listener)
+    assert airflow.__version__

rowmonitor_airflow-0.1.0/tests/test_client.py

@@ -0,0 +1,23 @@
+import time
+
+from rowmonitor_airflow.client import emit_async, post_json
+
+
+def test_post_swallows_errors(monkeypatch):
+    monkeypatch.setenv("ROWMONITOR_URL", "https://example.invalid")
+    monkeypatch.setenv("ROWMONITOR_TOKEN", "rm_airflow_test")
+
+    def boom(request, timeout):
+        raise TimeoutError("down")
+
+    post_json("/api/airflow/v1/events", {"type": "dag_run"}, opener=boom)
+
+
+def test_emit_returns_while_the_send_blocks():
+    started = time.monotonic()
+
+    def slow(path, payload):
+        time.sleep(0.8)
+
+    emit_async("/api/airflow/v1/events", {"type": "dag_run"}, sender=slow)
+    assert time.monotonic() - started < 0.4

rowmonitor_airflow-0.1.0/tests/test_listener.py

@@ -0,0 +1,39 @@
+from rowmonitor_airflow import listener
+from rowmonitor_airflow.metadata import dag_run_payload
+
+
+class DagRun:
+    dag_id = "orders"
+    run_id = "manual__1"
+    run_type = "manual"
+    logical_date = None
+    execution_date = None
+    start_date = None
+    end_date = None
+
+
+class Task:
+    dag_id = "orders"
+    task_id = "load"
+    run_id = "manual__1"
+    try_number = 1
+    start_date = None
+    end_date = None
+
+
+def test_hooks_do_not_raise_and_do_not_forward_the_error(monkeypatch):
+    sent = []
+    monkeypatch.setattr(listener, "emit_async", lambda path, payload: sent.append((path, payload)))
+    listener.on_dag_run_running(DagRun(), msg="queued row 9")
+    listener.on_dag_run_success(dag_run=DagRun(), msg="ok", session=object())
+    listener.on_dag_run_failed(DagRun(), "failed")
+    listener.on_task_instance_failed(None, Task(), error="SELECT secret FROM t", session=object())
+    assert [item[0] for item in sent] == ["/api/airflow/v1/events"] * 4
+    blob = str(sent)
+    assert "secret" not in blob
+    assert "queued row" not in blob
+    assert sent[0][1]["state"] == dag_run_payload(DagRun(), "running")["state"]
+
+
+def test_starting_a_non_scheduler_does_not_raise():
+    listener.on_starting(component=object())

rowmonitor_airflow-0.1.0/tests/test_metadata.py

@@ -0,0 +1,68 @@
+from datetime import datetime, timedelta, timezone
+
+from rowmonitor_airflow.metadata import (
+    DAG_RUN_KEYS,
+    HEARTBEAT_KEYS,
+    TASK_KEYS,
+    dag_run_payload,
+    heartbeat_body,
+    snapshot_dag,
+    task_failure_payload,
+)
+
+
+class DagRun:
+    def __init__(self):
+        self.dag_id = "orders"
+        self.run_id = "scheduled__2026-11-12T05:00:00+00:00"
+        self.run_type = "scheduled"
+        self.logical_date = datetime(2026, 11, 12, 5, tzinfo=timezone.utc)
+        self.start_date = self.logical_date
+        self.end_date = self.logical_date + timedelta(minutes=2)
+        self.conf = {"password": "secret"}
+        self.note = "row 42"
+
+
+class Task:
+    def __init__(self):
+        self.dag_id = "orders"
+        self.task_id = "load"
+        self.run_id = "scheduled__2026-11-12T05:00:00+00:00"
+        self.try_number = 2
+        self.start_date = datetime(2026, 11, 12, 5, tzinfo=timezone.utc)
+        self.end_date = None
+        self.log = "SELECT secret FROM customers"
+
+
+def test_dag_run_payload_is_metadata_only():
+    payload = dag_run_payload(DagRun(), "success")
+    assert set(payload) == set(DAG_RUN_KEYS)
+    assert payload["state"] == "success"
+    assert payload["duration_ms"] == 120_000
+    blob = str(payload)
+    assert "secret" not in blob
+    assert "customers" not in blob
+
+
+def test_task_failure_does_not_include_the_exception_or_log():
+    payload = task_failure_payload(Task())
+    assert set(payload) == set(TASK_KEYS)
+    assert "secret" not in str(payload)
+    assert payload["state"] == "failed"
+
+
+def test_heartbeat_skips_paused_and_caps_the_list():
+    class Row:
+        def __init__(self, dag_id, paused):
+            self.dag_id = dag_id
+            self.is_paused = paused
+            self.timetable_description = "0 * * * *"
+            self.next_dagrun = datetime(2026, 11, 12, 6, tzinfo=timezone.utc)
+
+    rows = [Row(f"dag_{index}", False) for index in range(3)]
+    rows.append(Row("paused", True))
+    body = heartbeat_body(rows, limit=2)
+    assert body["truncated"] is True
+    assert len(body["dags"]) == 2
+    assert set(body["dags"][0]) == set(HEARTBEAT_KEYS)
+    assert snapshot_dag(Row("paused", True)) is None