webchanges 3.36.0__tar.gz → 3.37.0__tar.gz

{webchanges-3.36.0/webchanges.egg-info → webchanges-3.37.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: webchanges
-Version: 3.36.0
+Version: 3.37.0
 Summary: Web Changes Delivered. AI-Summarized. Totally Anonymous.
 Author-email: Mike Borsetti <mike+webchanges@borsetti.com>
 Maintainer-email: Mike Borsetti <mike+webchanges@borsetti.com>
@@ -76,9 +76,9 @@ Provides-Extra: pypdf-crypto
 Requires-Dist: pypdf; extra == "pypdf-crypto"
 Requires-Dist: cryptography; extra == "pypdf-crypto"
 Provides-Extra: deepdiff
-Requires-Dist: deepdiff; extra == "deepdiff"
+Requires-Dist: deepdiff>=8.0; extra == "deepdiff"
 Provides-Extra: deepdiff-xml
-Requires-Dist: deepdiff; extra == "deepdiff-xml"
+Requires-Dist: deepdiff>=8.0; extra == "deepdiff-xml"
 Requires-Dist: xmltodict; extra == "deepdiff-xml"
 Provides-Extra: imagediff
 Requires-Dist: Pillow; extra == "imagediff"
@@ -118,26 +118,35 @@ notifications and AI-powered summaries to your favorite `platform
 <https://webchanges.readthedocs.io/en/stable/introduction.html#reporters>`__.
 
 
-Requirements
-============
-**webchanges** requires |support|.
+Highlights
+==========
+- **AI-powered change summaries** via Gemini (`ai_google differ
+  <https://webchanges.readthedocs.io/en/stable/differs.html#ai-google-diff>`__, BETA).
+- **Structural diffs for JSON and XML** via the `deepdiff differ
+  <https://webchanges.readthedocs.io/en/stable/differs.html#deepdiff-diff>`__.
+- **Visual diffs for images** via the `image differ
+  <https://webchanges.readthedocs.io/en/stable/differs.html#image-diff>`__ (BETA).
+- **Bypass TLS/JA3 fingerprinting** with the optional `curl_cffi
+  <https://webchanges.readthedocs.io/en/stable/jobs.html#http-client>`__ HTTP client.
+- **JSON-Schema-validated** ``jobs.yaml`` and ``config.yaml`` for editor autocompletion and inline validation.
 
-For the best experience, use the current version of `Python <https://www.python.org/downloads/>`__. We also support
-older Python versions for 3 years after they're replaced by a newer one; we just ask that you use the most up-to-date
-bug and security fix release from that older version.
 
-While **webchanges** supports free-threated Python, certain optional dependencies may not.
+Requirements
+============
+**webchanges** requires |support|. For the best experience, use the current version of `Python
+<https://www.python.org/downloads/>`__; older Python versions are supported for 3 years after they're replaced. Free-threaded Python is supported, though some
+optional dependencies may not be.
 
 For Generative AI summaries (BETA), you need a free `API Key from Google Cloud AI Studio
 <https://aistudio.google.com/app/apikey>`__ (see `here
-<https://webchanges.readthedocs.io/en/stable/differs.html#ai-google>`__).
+<https://webchanges.readthedocs.io/en/stable/differs.html#ai-google-diff>`__).
 
 
 Installation
 ============
 |pypi_version| |format| |status|
 
-Install **webchanges**  with ``uv`` (recommended):
+Install **webchanges**  with `uv <https://docs.astral.sh/uv/>`__ (recommended):
 
 .. code-block:: bash
 
@@ -149,22 +158,18 @@ or with ``pip``:
 
    pip install webchanges
 
-Running in Docker
+Other ways to run
 -----------------
-**webchanges** can easily run in a `Docker <https://www.docker.com/>`__ container! You will find a minimal
-implementation (no browser) `here <https://github.com/yubiuser/webchanges-docker>`__, and one with a browser
-`here <https://github.com/jhedlund/webchanges-docker>`__.
-
-As a GitHub Action
-------------------
-**webchanges** can easily run as a `GitHub Action <https://www.docker.com/>`__! You will find an implementation
-`here <https://github.com/swimmwatch/webchanges-action>`__.
+- In a `Docker <https://www.docker.com/>`__ container: a minimal implementation (no browser) is `here
+  <https://github.com/yubiuser/webchanges-docker>`__, and one with a browser is `here
+  <https://github.com/jhedlund/webchanges-docker>`__.
+- As a `GitHub Action <https://docs.github.com/en/actions>`__: an implementation is `here
+  <https://github.com/swimmwatch/webchanges-action>`__.
 
 
 Documentation |readthedocs|
 ===========================
-The documentation is hosted on `Read the Docs <https://webchanges.readthedocs.io/>`__.  A (very good) AI-generated
-high-level architectural overview is available on `DeepWiki <https://deepwiki.com/mborsetti/webchanges>`__.
+The documentation is hosted on `Read the Docs <https://webchanges.readthedocs.io/>`__.
 
 
 Quick Start
@@ -238,13 +243,11 @@ licensed under a `BSD 3-Clause License
 <https://raw.githubusercontent.com/thp/urlwatch/346b25914b0418342ffe2fb0529bed702fddc01f/COPYING>`__).
 
 
-Compatibility with **urlwatch**
-===============================
-
-This project is based on code from `urlwatch 2.21
-<https://github.com/thp/urlwatch/tree/346b25914b0418342ffe2fb0529bed702fddc01f>`__ dated 30 July 2020.
-For upgrading from **urlwatch** to **webchanges** (using the same job and
-configuration files), see `here <https://webchanges.readthedocs.io/en/stable/upgrading.html>`__.
+Relationship with **urlwatch**
+==============================
+This project is a fork of `urlwatch 2.21
+<https://github.com/thp/urlwatch/tree/346b25914b0418342ffe2fb0529bed702fddc01f>`__ and has since added many features;
+see the `comparison <https://webchanges.readthedocs.io/en/stable/_static/webchanges-vs-urlwatch.html>`__.
 
 
 .. |support| image:: https://img.shields.io/pypi/pyversions/webchanges.svg

{webchanges-3.36.0 → webchanges-3.37.0}/README.rst

@@ -14,26 +14,35 @@ notifications and AI-powered summaries to your favorite `platform
 <https://webchanges.readthedocs.io/en/stable/introduction.html#reporters>`__.
 
 
-Requirements
-============
-**webchanges** requires |support|.
+Highlights
+==========
+- **AI-powered change summaries** via Gemini (`ai_google differ
+  <https://webchanges.readthedocs.io/en/stable/differs.html#ai-google-diff>`__, BETA).
+- **Structural diffs for JSON and XML** via the `deepdiff differ
+  <https://webchanges.readthedocs.io/en/stable/differs.html#deepdiff-diff>`__.
+- **Visual diffs for images** via the `image differ
+  <https://webchanges.readthedocs.io/en/stable/differs.html#image-diff>`__ (BETA).
+- **Bypass TLS/JA3 fingerprinting** with the optional `curl_cffi
+  <https://webchanges.readthedocs.io/en/stable/jobs.html#http-client>`__ HTTP client.
+- **JSON-Schema-validated** ``jobs.yaml`` and ``config.yaml`` for editor autocompletion and inline validation.
 
-For the best experience, use the current version of `Python <https://www.python.org/downloads/>`__. We also support
-older Python versions for 3 years after they're replaced by a newer one; we just ask that you use the most up-to-date
-bug and security fix release from that older version.
 
-While **webchanges** supports free-threated Python, certain optional dependencies may not.
+Requirements
+============
+**webchanges** requires |support|. For the best experience, use the current version of `Python
+<https://www.python.org/downloads/>`__; older Python versions are supported for 3 years after they're replaced. Free-threaded Python is supported, though some
+optional dependencies may not be.
 
 For Generative AI summaries (BETA), you need a free `API Key from Google Cloud AI Studio
 <https://aistudio.google.com/app/apikey>`__ (see `here
-<https://webchanges.readthedocs.io/en/stable/differs.html#ai-google>`__).
+<https://webchanges.readthedocs.io/en/stable/differs.html#ai-google-diff>`__).
 
 
 Installation
 ============
 |pypi_version| |format| |status|
 
-Install **webchanges**  with ``uv`` (recommended):
+Install **webchanges**  with `uv <https://docs.astral.sh/uv/>`__ (recommended):
 
 .. code-block:: bash
 
@@ -45,22 +54,18 @@ or with ``pip``:
 
    pip install webchanges
 
-Running in Docker
+Other ways to run
 -----------------
-**webchanges** can easily run in a `Docker <https://www.docker.com/>`__ container! You will find a minimal
-implementation (no browser) `here <https://github.com/yubiuser/webchanges-docker>`__, and one with a browser
-`here <https://github.com/jhedlund/webchanges-docker>`__.
-
-As a GitHub Action
-------------------
-**webchanges** can easily run as a `GitHub Action <https://www.docker.com/>`__! You will find an implementation
-`here <https://github.com/swimmwatch/webchanges-action>`__.
+- In a `Docker <https://www.docker.com/>`__ container: a minimal implementation (no browser) is `here
+  <https://github.com/yubiuser/webchanges-docker>`__, and one with a browser is `here
+  <https://github.com/jhedlund/webchanges-docker>`__.
+- As a `GitHub Action <https://docs.github.com/en/actions>`__: an implementation is `here
+  <https://github.com/swimmwatch/webchanges-action>`__.
 
 
 Documentation |readthedocs|
 ===========================
-The documentation is hosted on `Read the Docs <https://webchanges.readthedocs.io/>`__.  A (very good) AI-generated
-high-level architectural overview is available on `DeepWiki <https://deepwiki.com/mborsetti/webchanges>`__.
+The documentation is hosted on `Read the Docs <https://webchanges.readthedocs.io/>`__.
 
 
 Quick Start
@@ -134,13 +139,11 @@ licensed under a `BSD 3-Clause License
 <https://raw.githubusercontent.com/thp/urlwatch/346b25914b0418342ffe2fb0529bed702fddc01f/COPYING>`__).
 
 
-Compatibility with **urlwatch**
-===============================
-
-This project is based on code from `urlwatch 2.21
-<https://github.com/thp/urlwatch/tree/346b25914b0418342ffe2fb0529bed702fddc01f>`__ dated 30 July 2020.
-For upgrading from **urlwatch** to **webchanges** (using the same job and
-configuration files), see `here <https://webchanges.readthedocs.io/en/stable/upgrading.html>`__.
+Relationship with **urlwatch**
+==============================
+This project is a fork of `urlwatch 2.21
+<https://github.com/thp/urlwatch/tree/346b25914b0418342ffe2fb0529bed702fddc01f>`__ and has since added many features;
+see the `comparison <https://webchanges.readthedocs.io/en/stable/_static/webchanges-vs-urlwatch.html>`__.
 
 
 .. |support| image:: https://img.shields.io/pypi/pyversions/webchanges.svg

{webchanges-3.36.0 → webchanges-3.37.0}/pyproject.toml

@@ -65,7 +65,7 @@ webchanges = 'webchanges.cli:main'
 use_browser = ['playwright', 'psutil']
 # filters
 # lxml is now optional with beautifulsoup4
-# note: also update the list of all possible dependencies in show_detailed_versions() in command.py!
+# note: also update the list of all possible dependencies in show_detailed_versions() in cli.py!
 beautify = ['beautifulsoup4', 'jsbeautifier', 'cssbeautifier']
 bs4 = ['beautifulsoup4']
 html5lib = ['html5lib']
@@ -76,8 +76,9 @@ pdf2text = ['pdftotext']
 pypdf = ['pypdf']
 pypdf_crypto = ['pypdf', 'cryptography']
 # differs
-deepdiff = ['deepdiff']
-deepdiff_xml = ['deepdiff', 'xmltodict']
+# deepdiff >= 8.0 is required for threshold_to_diff_deeper
+deepdiff = ['deepdiff >= 8.0']
+deepdiff_xml = ['deepdiff >= 8.0', 'xmltodict']
 imagediff = ['Pillow']
 # reporters
 matrix = ['matrix_client']
@@ -210,6 +211,14 @@ report_level = 'WARNING'
 # log_cli = true
 minversion = '9.0.2'
 testpaths = ['tests']
+filterwarnings = [
+    # Fail tests that leak sqlite3 connections. Scoped by message so third-party ResourceWarnings (e.g.
+    # asyncio/playwright transports) don't destabilize the suite. The second entry is required because the warning
+    # is emitted during garbage collection: the error raised there is unraisable, and only surfaces as a test
+    # failure through pytest's unraisable-exception plugin (as a PytestUnraisableExceptionWarning).
+    'error:unclosed database:ResourceWarning',
+    'error:(?s).*unclosed database:pytest.PytestUnraisableExceptionWarning',
+]
 
 # pytest-playwright
 # addopts = ['--browser', 'chromium', '--browser-channel', 'chrome']
@@ -293,7 +302,7 @@ ignore = [
     'RUF012',  # Mutable class attributes should be annotated with `typing.ClassVar`
     'SIM105',  # Use contextlib.suppress({exception}) instead of try-except-pass
     'SIM115',  # Use a context manager for opening files
-    'TD002',   #	Missing author in TODO
+    'TD002',   # Missing author in TODO
     'TD003',   # Missing issue link for this TODO
 ]
 

{webchanges-3.36.0 → webchanges-3.37.0}/tools/get_manpages_parser.py

@@ -1,8 +1,12 @@
-# pragma: exclude file
+from __future__ import annotations
 
 import sys
-from argparse import ArgumentParser
 from pathlib import Path
+from typing import TYPE_CHECKING
+
+if TYPE_CHECKING:
+    from argparse import ArgumentParser
+
 
 sys.path.insert(0, str(Path.cwd()))
 

{webchanges-3.36.0 → webchanges-3.37.0}/webchanges/__init__.py

@@ -22,7 +22,7 @@ __project_name__ = str(__package__)
 # * MINOR version when you add functionality in a backwards compatible manner, and
 # * MICRO or PATCH version when you make backwards compatible bug fixes. We no longer use '0'
 # If unsure on increments, use pkg_resources.parse_version to parse
-__version__ = '3.36.0'
+__version__ = '3.37.0'
 __description__ = (
     'Check web (or command output) for changes since last run and notify.\n\nAnonymously alerts you of web changes.'
 )

webchanges-3.37.0/webchanges/__main__.py

@@ -0,0 +1,11 @@
+if __name__ == '__main__':
+    import sys
+    from pathlib import Path
+
+    cwd = str(Path.cwd())
+    if cwd not in sys.path:
+        sys.path.insert(0, cwd)
+
+    from webchanges.cli import main
+
+    main()

{webchanges-3.36.0 → webchanges-3.37.0}/webchanges/_resources/config.schema.json

@@ -1,9 +1,10 @@
 {
   "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "https://raw.githubusercontent.com/mborsetti/webchanges/refs/heads/main/webchanges/_resources/config.schema.json",
   "title": "Webchanges Configuration File",
   "description": "JSON Schema for the webchanges config.yaml configuration file.",
   "type": "object",
-  "required": ["display", "report", "job_defaults", "differ_defaults", "database", "footnote"],
+  "required": ["display", "report", "job_defaults", "differ_defaults", "database", "footnote", "version_check"],
   "properties": {
     "display": {
       "$ref": "#/$defs/_ConfigDisplay",
@@ -29,6 +30,10 @@
       "type": ["string", "null"],
       "description": "Optional text appended to the footer of every report (text, html and markdown). Set to null to disable.",
       "default": null
+    },
+    "version_check": {
+      "$ref": "#/$defs/_ConfigVersionCheck",
+      "description": "Settings for the once-a-day check against PyPi for a newer release of webchanges."
     }
   },
   "additionalProperties": false,
@@ -37,7 +42,7 @@
       "title": "Display",
       "description": "Which job outcomes appear in reports. Changes are always reported.",
       "type": "object",
-      "required": ["new", "error", "unchanged", "empty-diff"],
+      "required": ["new", "error", "unchanged"],
       "properties": {
         "new": {
           "type": "boolean",
@@ -55,9 +60,8 @@
           "default": false
         },
         "empty-diff": {
-          "type": "boolean",
-          "description": "Deprecated. Whether to report a 'changed' job whose diff was reduced to an empty string by a diff_filter. Use the additions_only job directive instead.",
-          "default": false,
+          "enum": [false],
+          "description": "Deprecated; use additions_only in the job instead.",
           "deprecated": true
         }
       }
@@ -928,6 +932,19 @@
           "default": 4
         }
       }
+    },
+    "_ConfigVersionCheck": {
+      "title": "Version Check",
+      "description": "Settings for the once-a-day check against PyPi for a newer release of webchanges.",
+      "type": "object",
+      "required": ["enabled"],
+      "properties": {
+        "enabled": {
+          "type": "boolean",
+          "description": "Whether to query PyPi for a newer release. The result is cached on disk for 24 hours, so this hits the network at most once per day. Set to false to disable the check entirely (no network call, no upgrade nudge in reports).",
+          "default": true
+        }
+      }
     }
   }
 }

{webchanges-3.36.0 → webchanges-3.37.0}/webchanges/_resources/jobs.schema.json

@@ -1,7 +1,8 @@
 {
   "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "https://raw.githubusercontent.com/mborsetti/webchanges/refs/heads/main/webchanges/_resources/jobs.schema.json",
   "title": "Webchanges Jobs Configuration",
-  "description": "JSON Schema for each job in Webchanges' jobs.yaml configuration file",
+  "description": "JSON Schema for each job in webchanges' jobs.yaml configuration file",
   "type": "object",
   "oneOf": [{ "required": ["url"] }, { "required": ["command"] }],
   "properties": {
@@ -58,10 +59,23 @@
       "type": ["integer", "number"],
       "description": "Request timeout in seconds."
     },
+    "same_site_delay": {
+      "type": ["integer", "number"],
+      "description": "Number of seconds to wait before retrieving the URL when an earlier job in the list already targets the same site (network location). The first (or only) job for a given site is never delayed. Use it to stagger jobs that would otherwise hit the same site at the same instant and get rate-limited or blocked."
+    },
+    "delay": {
+      "type": ["integer", "number"],
+      "description": "Number of seconds to wait before retrieving the URL (e.g. to stagger jobs that hit the same site, or to space out requests to a slow or rate-limited server)."
+    },
     "ignore_connection_errors": {
       "type": "boolean",
       "description": "If true, connection errors are ignored (no report sent)."
     },
+    "empty_as_transient": {
+      "type": "boolean",
+      "description": "If true, an empty response is treated as a transient HTTP error (synthetic status code 999) instead of valid content, preserving the last non-empty snapshot. Combine with max_tries or 'ignore_http_error_codes: 999' to control error reporting. Not for jobs with use_browser.",
+      "default": false
+    },
     "no_redirects": {
       "type": "boolean",
       "description": "If true, do not follow HTTP redirects."
@@ -170,6 +184,11 @@
       },
       "description": "Resource types to block from loading (browser jobs)."
     },
+    "connect_over_cdp": {
+      "oneOf": [{ "type": "boolean" }, { "type": "string", "format": "uri" }],
+      "description": "Attach Playwright to an already-running Chromium browser via the Chrome DevTools Protocol (browser jobs). 'true' uses the default endpoint ws://127.0.0.1:58489/devtools/browser; a string is used as the CDP WebSocket URL.",
+      "default": false
+    },
     "evaluate": {
       "type": "string",
       "description": "JavaScript code to execute in the page context (browser jobs)."
@@ -180,7 +199,7 @@
     },
     "ignore_default_args": {
       "oneOf": [{ "type": "boolean" }, { "type": "array", "items": { "type": "string" } }],
-      "description": "Override Chrome default arguments (browser jobs).",
+      "description": "Override browser default arguments (browser jobs).",
       "default": false
     },
     "ignore_https_errors": {
@@ -204,14 +223,18 @@
       "type": "string",
       "description": "Referer header value (browser jobs)."
     },
+    "stealth": {
+      "type": "boolean",
+      "description": "Use stealth mode (browser jobs)."
+    },
     "switches": {
       "type": "array",
       "items": { "type": "string" },
-      "description": "Additional Chrome command-line switches (browser jobs)."
+      "description": "Additional browser command-line switches (browser jobs)."
     },
     "user_data_dir": {
       "type": "string",
-      "description": "Path to a Chrome user data directory (for persistence/auth with use_browser)."
+      "description": "Path to a browser user data directory (for persistence/auth with use_browser)."
     },
     "wait_for_function": {
       "oneOf": [
@@ -444,6 +467,27 @@
                   "type": "boolean",
                   "description": "Use compact output format.",
                   "default": false
+                },
+                "cutoff_distance_for_pairs": {
+                  "type": "number",
+                  "minimum": 0,
+                  "maximum": 1,
+                  "description": "With ignore_order, two changed items are paired and compared in depth only if their deep distance is below this value; otherwise they are reported as one item removed plus one added (deepdiff library's own default is 0.3).",
+                  "default": 1.0
+                },
+                "cutoff_intersection_for_pairs": {
+                  "type": "number",
+                  "minimum": 0,
+                  "maximum": 1,
+                  "description": "With ignore_order, pairing of changed items is attempted only if the intersection distance of the two iterables is below this value; otherwise each changed item is reported in its entirety (deepdiff library's own default is 0.7).",
+                  "default": 1.0
+                },
+                "threshold_to_diff_deeper": {
+                  "type": "number",
+                  "minimum": 0,
+                  "maximum": 1,
+                  "description": "The fraction of keys two dictionaries must share for them to be compared key by key instead of being reported as one whole new value (deepdiff library's own default is 0.33).",
+                  "default": 0
                 }
               },
               "required": ["name"],
@@ -644,21 +688,9 @@
   },
   "definitions": {
     "filterList": {
-      "oneOf": [
-        {
-          "type": "string",
-          "description": "A single filter name (e.g. 'html2text', 'sha1sum')."
-        },
-        {
-          "type": "array",
-          "items": { "$ref": "#/definitions/filterItem" },
-          "description": "List of filters to apply in order."
-        },
-        {
-          "$ref": "#/definitions/filterItemObject",
-          "description": "A single filter as an object."
-        }
-      ]
+      "type": "array",
+      "items": { "$ref": "#/definitions/filterItem" },
+      "description": "List of filters to apply in order."
     },
     "filterItem": {
       "oneOf": [
@@ -735,6 +767,21 @@
           ],
           "description": "Beautifies HTML using BeautifulSoup."
         },
+        "between": {
+          "type": "object",
+          "properties": {
+            "start": {
+              "type": "string",
+              "description": "Regular expression matching the line that opens the block (this line is not kept)."
+            },
+            "end": {
+              "type": "string",
+              "description": "Regular expression matching the line that closes the block (this line is not kept)."
+            }
+          },
+          "additionalProperties": false,
+          "description": "Extracts the block of lines between a starting and an ending regular expression."
+        },
         "css": {
           "oneOf": [
             { "type": "string", "description": "CSS selector expression." },

{webchanges-3.36.0 → webchanges-3.37.0}/webchanges/_vendored/headers.py

@@ -182,7 +182,7 @@ class Headers(typing.MutableMapping[str, str]):
         """
         return [(key.decode(self.encoding), value.decode(self.encoding)) for _, key, value in self._list]
 
-    def get(self, key: str, default: typing.Any = None) -> typing.Any:
+    def get(self, key: str, default: typing.Any = None) -> typing.Any:  # ty:ignore[invalid-method-override]
         """
         Return a header value. If multiple occurrences of the header occur
         then concatenate them together with commas.