watchdiff-core 0.2.2__tar.gz → 0.2.4__tar.gz

{watchdiff_core-0.2.2 → watchdiff_core-0.2.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: watchdiff-core
-Version: 0.2.2
+Version: 0.2.4
 Summary: Lightweight web change monitoring library - clean diffs, AI summaries, file/API/SSL/DB monitoring, structured alerts.
 Author: WatchDiff Contributors
 License: BSD-2-Clause
@@ -91,6 +91,12 @@ WatchDiff watches web pages, files, APIs, databases, and SSL certificates - then
 | Confirm change before alerting | `confirm_after=30` (flapping detection) |
 | Monitor a sitemap for URL changes | `.watch_sitemap("https://example.com/sitemap.xml")` |
 | Watch one JSON field only | `json_path="$.data.price"` |
+| Monitor authenticated pages | `cookies={"session": "abc123", "csrftoken": "xyz"}` |
+| Give a watcher a stable ID | `id="product-price"` — use ID instead of URL in `.pause()`/`.resume()` |
+| Limit concurrent checks | `WatchDiff(concurrency=5)` — cap parallel fetch workers |
+| Skip checks during downtime | `maintenance_windows=[MaintenanceWindow(from_="2026-01-15T02:00Z", to="2026-01-15T04:00Z")]` |
+| Restrict checks to business hours | `active_between=ActiveBetween(from_="09:00", to="17:00", days=[0,1,2,3,4], timezone="Europe/Paris")` |
+| Suppress noisy error alerts | `failure_policy=FailurePolicy(consecutive_failures=3, recovery_checks=2)` |
 | Send email on change | `alert=AlertConfig(email=EmailConfig(to="...", smtp=SmtpConfig(...)))` |
 | AI summary of changes | `ai_summary=True, ai_provider=AiProvider(type="gemini", api_key="...")` |
 | Customize the AI prompt | `ai_prompt="Summarize in French."` or `ai_prompt=lambda r: ...` |
@@ -146,7 +152,13 @@ WatchDiff watches web pages, files, APIs, databases, and SSL certificates - then
   - [Flapping detection](#flapping-detection)
   - [Sitemap monitoring](#sitemap-monitoring)
   - [JSON path targeting](#json-path-targeting)
+  - [Cookie-based authentication](#cookie-based-authentication)
   - [Email alerts](#email-alerts)
+  - [Watcher ID](#watcher-id)
+  - [Concurrency limiting](#concurrency-limiting)
+  - [Maintenance windows](#maintenance-windows)
+  - [Active hours](#active-hours)
+  - [Failure policy](#failure-policy)
 - [API reference](#api-reference)
   - [`.watch()`](#watchurl--)
   - [`.watch_db()`](#watch_dbconnection_string-table--)
@@ -154,8 +166,13 @@ WatchDiff watches web pages, files, APIs, databases, and SSL certificates - then
   - [`.watch_api()`](#watch_apiurl--)
   - [`.watch_cert()`](#watch_certhost--port443-warning_days30-)
   - [`.watch_sitemap()`](#watch_sitemapurl--)
-  - [`.start()` / `start_async()`](#startblock--startasync)
-  - [`.stop()` / pause / resume / status](#stop--pause--resume--status)
+  - [`.on_change()`](#on_changecallback)
+  - [`.start()` / `start_async()` / `.stop()`](#startblocktrue--await-start_async--stop)
+  - [`.check_once()`](#check_onceurl)
+  - [`.compare_urls()`](#compare_urlsurl_a-url_b--)
+  - [`.start_status_server()` / `.stop_status_server()`](#start_status_serverport-host--stop_status_server)
+  - [`.pause()` / `.resume()` / `.status()` / `.db_status()`](#pauseurl--resumeurl--status--db_status)
+  - [`.history()` / `.reports()` / `.clear()`](#historyurl--reportsurl--clearurl)
   - [`DiffReport`](#diffreport)
   - [`Change`](#change)
   - [`Snapshot`](#snapshot)
@@ -163,6 +180,14 @@ WatchDiff watches web pages, files, APIs, databases, and SSL certificates - then
   - [`SilenceInfo`](#silenceinfo)
   - [`AlertConfig`](#alertconfig)
   - [`BrowserOptions`](#browseroptions)
+  - [`SpikeInfo`](#spikeinfo)
+  - [`StatusChangeInfo`](#statuschangeinfo)
+  - [`StatusServer`](#statusserver)
+  - [`EmailConfig` / `SmtpConfig`](#emailconfig--smtpconfig)
+  - [`SitemapDiffReport` / `SitemapEntry`](#sitemapdifreport--sitemapentry)
+  - [`MaintenanceWindow`](#maintenancewindow)
+  - [`ActiveBetween`](#activebetween)
+  - [`FailurePolicy`](#failurepolicy)
   - [`DbDiffReport`](#dbdiffreport)
   - [`DbChange`](#dbchange)
   - [`DbWatcherStatus`](#dbwatcherstatus)
@@ -295,6 +320,9 @@ watchdiff db "postgresql://user:pass@localhost/mydb" products \
 # Generate a config file
 watchdiff init
 
+# Validate a config file without starting watchers
+watchdiff validate watchdiff.config.json
+
 # Run from config file
 watchdiff run --config watchdiff.config.json
 
@@ -1087,7 +1115,15 @@ Edit `watchdiff.config.json`:
       "ignore_selectors": [".cookie-banner", "#ad-container"],
       "ignore_patterns": ["\\d+ views"],
       "timeout": 15,
-      "headers": {}
+      "headers": {},
+      "schedule": null,
+      "confirm_after": null,
+      "json_path": null,
+      "email": null,
+      "id": null,
+      "maintenance_windows": [],
+      "active_between": null,
+      "failure_policy": null
     },
     {
       "url": "https://hnrss.org/frontpage",
@@ -1395,36 +1431,218 @@ value = extract_json_path('{"data": {"price": 42.5}}', "$.data.price")
 # "42.5"
 ```
 
+## Cookie-based authentication
+
+Pass cookies with every request to monitor pages that require a login session. Cookies are merged into the outgoing `Cookie` header on top of any `headers` you supply.
+
+```python
+wd.watch("https://app.example.com/dashboard",
+         cookies={"session": "your-session-token", "csrftoken": "your-csrf-token"},
+         interval=300,
+         on_change=lambda r: print(r.changes))
+```
+
+Combine `cookies` with `headers` — the cookie string is appended to any existing `Cookie` header:
+
+```python
+wd.watch_api("https://api.example.com/account",
+             headers={"Authorization": "Bearer token123"},
+             cookies={"_ga": "GA1.1.0000000000.0000000000"},
+             json_path="$.balance",
+             interval=60)
+```
+
+> Tip: copy cookie values from your browser's DevTools → Network tab → Request Headers.
+
 ## Email alerts
 
 Send email notifications via SMTP when a change is detected. Uses Python stdlib `smtplib` — no extra dependency required.
 
+### Quickstart — `email=` shortcut
+
+Pass an `EmailConfig` directly to `.watch()`:
+
 ```python
 from watchdiff import WatchDiff
-from watchdiff.models import AlertConfig, EmailConfig, SmtpConfig
+from watchdiff.models import EmailConfig, SmtpConfig
 
 wd = WatchDiff()
+wd.watch(
+    "https://example.com/prices",
+    email=EmailConfig(
+        to="alerts@example.com",
+        smtp=SmtpConfig(
+            host="smtp.gmail.com",
+            port=465,
+            user="you@gmail.com",
+            password="app-password",
+        ),
+    ),
+)
+wd.start()
+```
+
+### Via `AlertConfig` — combine with webhooks and callbacks
+
+```python
+from watchdiff.models import AlertConfig, EmailConfig, SmtpConfig
+
 wd.watch("https://example.com/prices",
          alert=AlertConfig(
-             on_change=[],
-             webhooks=[],
+             on_change=[lambda r: print(r.summary())],
+             webhooks=["https://ntfy.sh/my-topic"],
              min_changes=1,
              email=EmailConfig(
-                 to="alerts@example.com",
+                 to=["alerts@example.com", "team@example.com"],
+                 from_="watchdiff@example.com",
+                 subject="Price change detected",
                  smtp=SmtpConfig(
                      host="smtp.gmail.com",
-                     port=465,
+                     port=587,   # STARTTLS
                      user="you@gmail.com",
                      password="app-password",
                  ),
              ),
          ))
+```
+
+### Config file (`watchdiff.config.json`)
+
+```json
+{
+  "url": "https://example.com/prices",
+  "email": {
+    "to": "alerts@example.com",
+    "from": "watchdiff@example.com",
+    "subject": "Price change detected",
+    "smtp": {
+      "host": "smtp.gmail.com",
+      "port": 465,
+      "user": "you@gmail.com",
+      "password": "app-password"
+    }
+  }
+}
+```
+
+**Notes:**
+- `from_` defaults to `user@host` when omitted
+- `subject` defaults to `"[WatchDiff] Change detected: {label}"`
+- Multiple recipients: `to=["a@x.com", "b@x.com"]`
+- Port `465` uses SSL from the start (`SMTP_SSL`). All other ports use `STARTTLS`.
+
+## Watcher ID
+
+Assign a stable string identifier to any watcher. The ID is used instead of the URL for `.pause()`, `.resume()`, and `.status()` lookups, which is useful when two watchers share the same URL or when you want readable keys in your control code.
+
+```python
+from watchdiff import WatchDiff
+
+wd = WatchDiff()
+wd.watch("https://example.com/prices", id="product-price", interval=60)
+wd.watch("https://example.com/prices", id="product-price-2", interval=120)
+
+wd.pause("product-price")   # pause by ID, not URL
+wd.resume("product-price")
+```
+
+Without `id`, `pause(url)` pauses **all** watchers registered for that URL. With `id`, each watcher is independent and can be paused or resumed individually.
+
+IDs also appear in `WatcherStatus.id` and in the JSON returned by `status()`.
+
+## Concurrency limiting
+
+Cap the number of checks that run simultaneously. Without a limit all watchers fire in parallel; with a limit the scheduler queues excess checks.
+
+```python
+wd = WatchDiff(concurrency=5)  # at most 5 simultaneous fetches
+```
+
+Useful when you monitor hundreds of URLs and want to avoid hammering a shared proxy pool or saturating a network interface.
+
+## Maintenance windows
+
+Skip checks during a known downtime window. Checks resume automatically once the window closes.
+
+```python
+from watchdiff import WatchDiff, MaintenanceWindow
+
+wd = WatchDiff()
+wd.watch(
+    "https://api.example.com/health",
+    interval=60,
+    maintenance_windows=[
+        MaintenanceWindow(
+            from_="2026-02-01T02:00:00+00:00",  # ISO 8601 UTC start
+            to="2026-02-01T04:00:00+00:00",       # UTC end
+        ),
+    ],
+)
+wd.start()
+```
+
+- `from_` and `to` accept either an ISO 8601 string or a `datetime` object.
+- Multiple windows can be listed; any overlapping window suppresses the check.
+- The window is evaluated fresh on each tick — no restart needed after the window passes.
+- `WatcherStatus.in_maintenance` reflects the current state.
+
+## Active hours
+
+Restrict checks to specific hours and/or days. Checks outside the window are silently skipped until the next window opens.
+
+```python
+from watchdiff import WatchDiff, ActiveBetween
+
+wd = WatchDiff()
+wd.watch(
+    "https://example.com/prices",
+    interval=60,
+    active_between=ActiveBetween(
+        from_="09:00",   # "HH:MM" — start of active window
+        to="17:00",      # "HH:MM" — end of active window
+        days=[0, 1, 2, 3, 4],   # Mon–Fri (0=Monday, 6=Sunday). None = every day
+        timezone="Europe/Paris", # IANA timezone name. None = UTC
+    ),
+)
 wd.start()
 ```
 
-Optional fields: `from_` (defaults to `user@host`), `subject` (defaults to `"[WatchDiff] Change detected: {label}"`). Multiple recipients: `to=["a@x.com", "b@x.com"]`.
+- `to` before `from_` (e.g. `from_="22:00"`, `to="06:00"`) is treated as an overnight window.
+- `days` defaults to all 7 days when omitted. `timezone` defaults to UTC when omitted.
+- Requires Python 3.9+ `zoneinfo` stdlib. Falls back to UTC if the IANA database is unavailable.
+- **Note:** the Python API uses integer weekday indices (`0`=Monday … `6`=Sunday, matching `datetime.weekday()`). The TypeScript port uses string names (`"monday"`, `"friday"`, …).
 
-Port `465` uses SSL from the start (`SMTP_SSL`). Other ports use `STARTTLS`.
+## Failure policy
+
+Gate the `on_error` callback until a URL has failed a configurable number of times in a row, and require a configurable number of consecutive successes before considering it recovered. Reduces alert noise from transient blips.
+
+```python
+from watchdiff import WatchDiff, FailurePolicy
+
+wd = WatchDiff()
+wd.watch(
+    "https://example.com/status",
+    interval=30,
+    failure_policy=FailurePolicy(
+        consecutive_failures=3,   # only fire on_error after 3 failures in a row
+        recovery_checks=2,        # require 2 consecutive successes to clear the failure state
+        respect_retry_after=True, # honour Retry-After response header when present
+    ),
+    on_error=lambda exc, cfg: print(f"Confirmed failure: {exc}"),
+)
+wd.start()
+```
+
+Without `failure_policy`, the first fetch error fires `on_error` immediately. With it:
+
+- Errors below the threshold are silently swallowed — no callback, no alert.
+- After `consecutive_failures` errors in a row, `on_error` fires **once**, then is suppressed until recovery.
+- Recovery requires `recovery_checks` consecutive successful fetches.
+- When `respect_retry_after=True` and the server returns a `Retry-After` header (e.g. on 429/503), the next check is postponed until the header deadline.
+
+Default: `consecutive_failures=3`, `recovery_checks=1`, `respect_retry_after=False`.
+
+> **Note:** the TypeScript port defaults `consecutiveFailures` to **1** (fire on the first error). Python defaults to **3**. A bare `FailurePolicy()` therefore behaves differently across the two implementations.
 
 ## API reference
 
@@ -1437,8 +1655,15 @@ from watchdiff.store import SqliteStore
 wd = WatchDiff()                               # JSON store in .watchdiff/
 wd = WatchDiff(storage_dir="/data/watchdiff")  # custom JSON store path
 wd = WatchDiff(store=SqliteStore("db.sqlite")) # SQLite store
+wd = WatchDiff(concurrency=5)                  # at most 5 parallel fetch workers
 ```
 
+| Parameter | Type | Default | Description |
+|---|---|---|---|
+| `storage_dir` | `str` | `".watchdiff"` | Directory for JSON snapshot/report files |
+| `store` | `Store \| None` | `None` | Custom store implementation (e.g. `SqliteStore`) |
+| `concurrency` | `int \| None` | `None` | Max simultaneous check workers. `None` = unlimited |
+
 #### `.watch(url, *, ...)`
 
 Register a URL to monitor. All keyword arguments are optional. Returns `self` (chainable).
@@ -1480,6 +1705,7 @@ Register a URL to monitor. All keyword arguments are optional. Returns `self` (c
 | `on_spike` | `Callable \| None` | `None` | Called with `SpikeInfo` when spike is detected |
 | `alert_on_status_change` | `bool` | `False` | Alert when HTTP status code changes (200→503, etc.) |
 | `on_status_change` | `Callable \| None` | `None` | Called with `StatusChangeInfo` on status code change |
+| `cookies` | `dict[str, str]` | `{}` | Cookies sent with every request — merged into the `Cookie` header (e.g. `{"session": "abc"}`) |
 | `alert` | `AlertConfig \| None` | `None` | Full alert config — use instead of `on_change`/`webhooks` to include email or fine-tune retries |
 | `alert_if` | `Callable[[DiffReport], bool] \| None` | `None` | Custom gate - only alert when this function returns `True` |
 | `expected_status` | `int \| None` | `None` | Fire `on_error` when actual HTTP status differs from this value |
@@ -1490,6 +1716,11 @@ Register a URL to monitor. All keyword arguments are optional. Returns `self` (c
 | `schedule` | `str \| None` | `None` | 5-field cron expression. Overrides `interval` when set. |
 | `confirm_after` | `int \| None` | `None` | Re-verify after N seconds before alerting (flapping detection) |
 | `json_path` | `str \| None` | `None` | JSON path expression to extract a sub-value before diffing (e.g. `"$.data.price"`) |
+| `email` | `EmailConfig \| None` | `None` | SMTP email alert fired on every detected change — shortcut over building a full `AlertConfig` |
+| `id` | `str \| None` | `None` | Stable identifier for this watcher — used instead of URL in `.pause()`/`.resume()` |
+| `maintenance_windows` | `list[MaintenanceWindow]` | `[]` | One-time UTC time ranges during which checks are skipped |
+| `active_between` | `ActiveBetween \| None` | `None` | Restrict checks to a recurring daily/weekly time window |
+| `failure_policy` | `FailurePolicy \| None` | `None` | Gate `on_error` until N consecutive failures; require M consecutive successes to recover |
 
 ```python
 # Chainable
@@ -1797,6 +2028,8 @@ status.checks_count    # int
 status.changes_count   # int
 status.errors_count    # int
 status.last_status_code # int — last known HTTP status (0 = unknown)
+status.id               # str | None — stable watcher ID (set via id= on .watch())
+status.in_maintenance   # bool — True if currently inside a maintenance window
 
 status.as_dict()       # JSON-serialisable dict
 ```
@@ -1865,6 +2098,128 @@ server.start()
 server.stop()
 ```
 
+### `EmailConfig` / `SmtpConfig`
+
+```python
+from watchdiff import EmailConfig, SmtpConfig
+
+email = EmailConfig(
+    to="ops@example.com",          # str or list[str]
+    smtp=SmtpConfig(
+        host="smtp.example.com",
+        port=587,
+        user="alerts@example.com",
+        password="secret",
+        secure=None,               # None = auto (SSL on 465, STARTTLS otherwise)
+    ),
+    from_="alerts@example.com",    # optional sender address
+    subject="[WatchDiff] change",  # optional subject override
+)
+```
+
+Pass as `email=` to `.watch()` or inside `AlertConfig(email=...)`.
+
+---
+
+### `SitemapDiffReport` / `SitemapEntry`
+
+Passed to callbacks registered with `.watch_sitemap()`:
+
+```python
+report.sitemap_url   # str — URL of the sitemap
+report.label         # str — human-readable label
+report.added         # list[SitemapEntry] — URLs newly present in the sitemap
+report.removed       # list[SitemapEntry] — URLs no longer in the sitemap
+report.compared_at   # datetime — UTC timestamp of the comparison
+```
+
+Each `SitemapEntry`:
+
+```python
+entry.url            # str
+entry.last_modified  # str | None — <lastmod> value from the sitemap
+entry.change_freq    # str | None — <changefreq> value
+entry.priority       # str | None — <priority> value
+```
+
+---
+
+### `MaintenanceWindow`
+
+Defines a one-time UTC window during which checks are paused:
+
+```python
+from watchdiff import MaintenanceWindow
+from datetime import datetime, timezone
+
+# From ISO 8601 strings (recommended)
+w = MaintenanceWindow(
+    from_="2026-02-01T02:00:00+00:00",
+    to="2026-02-01T04:00:00+00:00",
+)
+
+# From datetime objects
+w = MaintenanceWindow(
+    from_=datetime(2026, 2, 1, 2, 0, tzinfo=timezone.utc),
+    to=datetime(2026, 2, 1, 4, 0, tzinfo=timezone.utc),
+)
+```
+
+| Field | Type | Description |
+|---|---|---|
+| `from_` | `datetime \| str` | UTC start; ISO 8601 string or `datetime` |
+| `to` | `datetime \| str` | UTC end |
+
+ISO strings are parsed in `__post_init__`. `"Z"` suffix is accepted as `+00:00`.
+
+---
+
+### `ActiveBetween`
+
+Restricts checks to a recurring daily or weekly time window:
+
+```python
+from watchdiff import ActiveBetween
+
+ab = ActiveBetween(
+    from_="09:00",
+    to="17:00",
+    days=[0, 1, 2, 3, 4],   # 0=Monday … 6=Sunday. None = every day
+    timezone="Europe/Paris", # IANA name. None = UTC
+)
+```
+
+| Field | Type | Default | Description |
+|---|---|---|---|
+| `from_` | `str` | — | Window start as `"HH:MM"` |
+| `to` | `str` | — | Window end as `"HH:MM"`. If before `from_`, treated as overnight |
+| `days` | `list[int] \| None` | `None` | Weekdays to restrict to (`0`=Mon). `None` = every day |
+| `timezone` | `str \| None` | `None` | IANA timezone name (e.g. `"America/New_York"`). `None` = UTC |
+
+---
+
+### `FailurePolicy`
+
+Gates `on_error` to avoid alert noise from transient blips:
+
+```python
+from watchdiff import FailurePolicy
+
+fp = FailurePolicy(
+    consecutive_failures=3,   # fire on_error only after this many consecutive failures
+    recovery_checks=2,        # require this many consecutive successes to clear failure state
+    respect_retry_after=True, # honour Retry-After response header if present
+)
+```
+
+| Field | Type | Default | Description |
+|---|---|---|---|
+| `consecutive_failures` | `int` | `3` | Minimum consecutive failures before `on_error` fires (TypeScript default: `1`) |
+| `recovery_checks` | `int` | `1` | Consecutive successes required to exit failure mode |
+| `respect_retry_after` | `bool` | `False` | Use `Retry-After` header delay for the next check |
+
+---
+
 ### `DbDiffReport`
 
 Returned by `on_change` and `DbDiffEngine.compare()`:
@@ -1968,6 +2323,7 @@ summary = db_report_summary(report) # "orders: 1 inserted"
 ```
 Commands:
   init      Generate a watchdiff.config.json template
+  validate  Check a config file for errors without starting any watchers
   run       Start continuous monitoring (URL or config file)
   db        Monitor a database table for changes
   compare   Fetch two URLs and compare their content
@@ -2002,11 +2358,18 @@ Options for run:
   --alert-if-no-change    Fire silence alert after N seconds without change (0 = off)
   --proxy                 Proxy URL (repeatable)
   --user-agent            User-Agent string (repeatable)
+  --schedule              5-field cron expression — overrides --interval when set
+  --confirm-after         Re-fetch after N seconds before confirming a change (0 = off)
+  --json-path             $.dot.path expression to extract from JSON response before diffing
   --webhook          -w   Webhook URL (repeatable)
   --log-format            Log format: text | json (default text)
   --verbose          -v   Enable debug logging
   --quiet            -q   Suppress change output
 
+Options for validate:
+  --json                  Output result as JSON (exit code 0 = valid, 1 = invalid)
+  --verbose          -v   Enable debug logging
+
 Options for db:
   --diff-mode        -m   row | schema | aggregate | value (default row)
   --interval         -i   Seconds between checks (default 60)
@@ -2086,6 +2449,9 @@ Every CLI option can be set via environment variable — useful for Docker, CI,
 | `WATCHDIFF_ALERT_IF_NO_CHANGE` | `--alert-if-no-change` | `86400` |
 | `WATCHDIFF_PROXY` | `--proxy` | `http://proxy:8080` |
 | `WATCHDIFF_USER_AGENT` | `--user-agent` | `MyBot/1.0` |
+| `WATCHDIFF_SCHEDULE` | `--schedule` | `0 9 * * *` |
+| `WATCHDIFF_CONFIRM_AFTER` | `--confirm-after` | `30` |
+| `WATCHDIFF_JSON_PATH` | `--json-path` | `$.data.price` |
 | `WATCHDIFF_TARGET` | `--target` | `.price` |
 | `WATCHDIFF_QUIET` | `--quiet` | `true` |
 | `WATCHDIFF_LOG_FORMAT` | `--log-format` | `json` |