webchanges 3.24.1__tar.gz → 3.26.0__tar.gz

{webchanges-3.24.1/webchanges.egg-info → webchanges-3.26.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: webchanges
-Version: 3.24.1
+Version: 3.26.0
 Summary: Check web (or command output) for changes since last run and notify. Anonymously alerts you of web changes, with
 Author-email: Mike Borsetti <mike+webchanges@borsetti.com>
 Maintainer-email: Mike Borsetti <mike+webchanges@borsetti.com>
@@ -80,6 +80,7 @@ Classifier: Environment :: Console
 Classifier: Topic :: Internet
 Classifier: Topic :: Internet :: WWW/HTTP
 Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
+Classifier: Topic :: System :: Monitoring
 Classifier: Topic :: Utilities
 Classifier: Development Status :: 5 - Production/Stable
 Classifier: License :: OSI Approved :: MIT License
@@ -87,17 +88,19 @@ Classifier: License :: OSI Approved :: BSD License
 Classifier: Programming Language :: Python
 Classifier: Programming Language :: Python :: 3
 Classifier: Programming Language :: Python :: 3 :: Only
-Classifier: Programming Language :: Python :: 3.9
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
 Classifier: Programming Language :: Python :: Implementation :: CPython
 Classifier: Operating System :: OS Independent
-Classifier: Natural Language :: English
+Classifier: Environment :: Console
 Classifier: Intended Audience :: End Users/Desktop
 Classifier: Intended Audience :: System Administrators
 Classifier: Intended Audience :: Developers
-Requires-Python: >=3.9
+Classifier: Natural Language :: English
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
 Description-Content-Type: text/x-rst
 License-File: LICENSE
 Requires-Dist: colorama; os_name == "nt"
@@ -157,8 +160,10 @@ Provides-Extra: requests
 Requires-Dist: requests; extra == "requests"
 Provides-Extra: safe-password
 Requires-Dist: keyring; extra == "safe-password"
+Provides-Extra: zstd
+Requires-Dist: zstandard; extra == "zstd"
 Provides-Extra: all
-Requires-Dist: webchanges[beautify,bs4,deepdiff_xml,html5lib,ical2text,imagediff,jq,matrix,ocr,pdf2text,pushbullet,pushover,pypdf_crypto,redis,requests,safe_password,use_browser,xmpp]; extra == "all"
+Requires-Dist: webchanges[beautify,bs4,deepdiff_xml,html5lib,ical2text,imagediff,jq,matrix,ocr,pdf2text,pushbullet,pushover,pypdf_crypto,redis,requests,safe_password,use_browser,xmpp,zstd]; extra == "all"
 
 .. role:: underline
     :class: underline
@@ -204,7 +209,7 @@ Install **webchanges** |pypi_version| |format| |status| |security| with:
 
 Running in Docker
 =================
-**webchanges** can be run in a `Docker <https://www.docker.com/>`__ container. Please see `here
+**webchanges** can be run in a `Docker <https://www.docker.com/>`__ container; please see `here
 <https://github.com/yubiuser/webchanges-docker>`__ for one such implementation.
 
 
@@ -245,14 +250,14 @@ execution, just run:
 
    webchanges
 
-**webchanges** does not include a scheduler. We recommend using a system scheduler to automatically run **webchanges**
+**webchanges** does not include a scheduler. We recommend using a system scheduler to automatically run it
 periodically:
 
 - On Linux or macOS, you can use cron (if you have never used cron before, see
   `here <https://www.computerhope.com/unix/ucrontab.htm>`__); `crontab.guru <https://crontab.guru>`__ will build a
-  schedule expression for you.
+  schedule expression for you;
 - On macOS, you can use `launchd <https://developer.apple
-  .com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/ScheduledJobs.html>`__
+  .com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/ScheduledJobs.html>`__;
 - On Windows, you can use the built-in `Windows Task Scheduler
   <https://en.wikipedia.org/wiki/Windows_Task_Scheduler>`__.
 
@@ -294,28 +299,28 @@ easily upgrade to **webchanges** from the current version of **urlwatch** using
 (see `here <https://webchanges.readthedocs.io/en/stable/migration.html>`__) and benefit from many improvements,
 including:
 
-* Summary of changes in plain text using Generative AI, useful for long, boring, legal documents;
+* Summary of changes in plain text using Generative AI, useful for long documents (e.g. legal);
 * Depicting changes to an image;
 * Element-by-element changes of JSON or XML data;
-* Much better `documentation <https://webchanges.readthedocs.io/>`__;
-* Many improvements to HTML reports, including:
+* Much better `documentation <https://webchanges.readthedocs.io/>`__, easing implementation;
+* Much clearer to HTML reports, which include:
 
   * Links that are `clickable <https://pypi.org/project/webchanges/>`__!
   * Retaining of original formatting such as **bolding / headers**, *italics*, :underline:`underlining`, list bullets
     (•) and indentation;
-  * :additions:`Added` and :deletions:`deleted` lines clearly highlighted by color and strikethrough, and long lines
-    that wrap around;
+  * Lines that are :additions:`added` and :deletions:`deleted` are clearly highlighted by color and strikethrough,
+    and long lines wrap around;
   * Correct rendering by email clients who override stylesheets (e.g. Gmail);
   * Other legibility improvements;
 
 * New filters such as `additions_only <https://webchanges.readthedocs.io/en/stable/diff_filters.html#additions-only>`__,
   which makes it easier to track content that was added without the distractions of the content that was deleted;
-* New command line arguments such as ``--errors`` to catch jobs that no longer work;
+* New command line arguments, such as ``--errors`` (to catch jobs that no longer work);
 * More reliability and stability, including a ~30 percentage point increase in testing coverage;
 * Many other additions, refinements and fixes (see `detailed information
   <https://webchanges.readthedocs.io/en/stable/migration.html#upgrade-details>`__).
 
-Examples:
+Example enhancements to HTML reporting:
 
 .. image:: https://raw.githubusercontent.com/mborsetti/webchanges/main/docs/html_diff_filters_example_1.png
     :width: 504

{webchanges-3.24.1 → webchanges-3.26.0}/README.rst

@@ -42,7 +42,7 @@ Install **webchanges** |pypi_version| |format| |status| |security| with:
 
 Running in Docker
 =================
-**webchanges** can be run in a `Docker <https://www.docker.com/>`__ container. Please see `here
+**webchanges** can be run in a `Docker <https://www.docker.com/>`__ container; please see `here
 <https://github.com/yubiuser/webchanges-docker>`__ for one such implementation.
 
 
@@ -83,14 +83,14 @@ execution, just run:
 
    webchanges
 
-**webchanges** does not include a scheduler. We recommend using a system scheduler to automatically run **webchanges**
+**webchanges** does not include a scheduler. We recommend using a system scheduler to automatically run it
 periodically:
 
 - On Linux or macOS, you can use cron (if you have never used cron before, see
   `here <https://www.computerhope.com/unix/ucrontab.htm>`__); `crontab.guru <https://crontab.guru>`__ will build a
-  schedule expression for you.
+  schedule expression for you;
 - On macOS, you can use `launchd <https://developer.apple
-  .com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/ScheduledJobs.html>`__
+  .com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/ScheduledJobs.html>`__;
 - On Windows, you can use the built-in `Windows Task Scheduler
   <https://en.wikipedia.org/wiki/Windows_Task_Scheduler>`__.
 
@@ -132,28 +132,28 @@ easily upgrade to **webchanges** from the current version of **urlwatch** using
 (see `here <https://webchanges.readthedocs.io/en/stable/migration.html>`__) and benefit from many improvements,
 including:
 
-* Summary of changes in plain text using Generative AI, useful for long, boring, legal documents;
+* Summary of changes in plain text using Generative AI, useful for long documents (e.g. legal);
 * Depicting changes to an image;
 * Element-by-element changes of JSON or XML data;
-* Much better `documentation <https://webchanges.readthedocs.io/>`__;
-* Many improvements to HTML reports, including:
+* Much better `documentation <https://webchanges.readthedocs.io/>`__, easing implementation;
+* Much clearer to HTML reports, which include:
 
   * Links that are `clickable <https://pypi.org/project/webchanges/>`__!
   * Retaining of original formatting such as **bolding / headers**, *italics*, :underline:`underlining`, list bullets
     (•) and indentation;
-  * :additions:`Added` and :deletions:`deleted` lines clearly highlighted by color and strikethrough, and long lines
-    that wrap around;
+  * Lines that are :additions:`added` and :deletions:`deleted` are clearly highlighted by color and strikethrough,
+    and long lines wrap around;
   * Correct rendering by email clients who override stylesheets (e.g. Gmail);
   * Other legibility improvements;
 
 * New filters such as `additions_only <https://webchanges.readthedocs.io/en/stable/diff_filters.html#additions-only>`__,
   which makes it easier to track content that was added without the distractions of the content that was deleted;
-* New command line arguments such as ``--errors`` to catch jobs that no longer work;
+* New command line arguments, such as ``--errors`` (to catch jobs that no longer work);
 * More reliability and stability, including a ~30 percentage point increase in testing coverage;
 * Many other additions, refinements and fixes (see `detailed information
   <https://webchanges.readthedocs.io/en/stable/migration.html#upgrade-details>`__).
 
-Examples:
+Example enhancements to HTML reporting:
 
 .. image:: https://raw.githubusercontent.com/mborsetti/webchanges/main/docs/html_diff_filters_example_1.png
     :width: 504

{webchanges-3.24.1 → webchanges-3.26.0}/pyproject.toml

@@ -19,7 +19,7 @@ description = """\
     Gen AI summaries (BETA).\
     """
 readme = { file = 'README.rst', content-type = 'text/x-rst' }
-requires-python = '>=3.9'
+requires-python = '>=3.10'
 license = {file = 'LICENSE'}
 authors = [
     {name = 'Mike Borsetti', email = 'mike+webchanges@borsetti.com'},
@@ -33,6 +33,7 @@ classifiers = [
     'Topic :: Internet',
     'Topic :: Internet :: WWW/HTTP',
     'Topic :: Internet :: WWW/HTTP :: Indexing/Search',
+    'Topic :: System :: Monitoring',
     'Topic :: Utilities',
     'Development Status :: 5 - Production/Stable',
     'License :: OSI Approved :: MIT License',
@@ -40,16 +41,18 @@ classifiers = [
     'Programming Language :: Python',
     'Programming Language :: Python :: 3',
     'Programming Language :: Python :: 3 :: Only',
-    'Programming Language :: Python :: 3.9',
     'Programming Language :: Python :: 3.10',
     'Programming Language :: Python :: 3.11',
     'Programming Language :: Python :: 3.12',
+    'Programming Language :: Python :: 3.13',
     'Programming Language :: Python :: Implementation :: CPython',
     'Operating System :: OS Independent',
-    'Natural Language :: English',
+    'Environment :: Console',
     'Intended Audience :: End Users/Desktop',
     'Intended Audience :: System Administrators',
     'Intended Audience :: Developers',
+    'Natural Language :: English',
+    'Typing :: Typed',
 ]
 
 [project.urls]
@@ -90,8 +93,9 @@ xmpp = ['aioxmpp']
 redis = ['redis']
 requests = ['requests']
 safe_password = ['keyring']
+zstd = ['zstandard']
 all = [
-    'webchanges[use_browser,beautify,bs4,html5lib,ical2text,jq,ocr,pdf2text,pypdf_crypto,deepdiff_xml,imagediff,matrix,pushbullet,pushover,xmpp,redis,requests,safe_password]'
+    'webchanges[use_browser,beautify,bs4,html5lib,ical2text,jq,ocr,pdf2text,pypdf_crypto,deepdiff_xml,imagediff,matrix,pushbullet,pushover,xmpp,redis,requests,safe_password,zstd]'
 ]
 
 
@@ -155,7 +159,7 @@ color_output = true
 [tool.black]
 # What's in here overrides the command-line options shown by running $ black --help.
 line_length = 120
-target_version = ['py39']
+target_version = ['py310']
 skip_string_normalization = true
 extend_exclude = '/(\.idea|\.pytest_cache|\__pycache__|\venv.*|\webchanges.egg-info)/'
 color = true
@@ -305,11 +309,12 @@ exclude_lines = [
 log_auto_indent = true
 # Enable log display during test run (aka "live logging" https://docs.pytest.org/en/stable/logging.html#live-logs)
 log_cli = true
-minversion = '7.4.0'
+minversion = '8.3.3'
 testpaths = ['tests']
 
 # the below is for pytest-asyncio (required due to Playwright)
 asyncio_mode = 'auto'
+asyncio_default_fixture_loop_scope = 'function'
 
 # Adds pytest-cov functionality (see https://pytest-cov.readthedocs.io/en/latest/config.html)
 # Note: --cov moved to .github/workflows/ci-cd.yaml and tox.ini due to interference with PyCharm breakpoints (see

{webchanges-3.24.1 → webchanges-3.26.0}/webchanges/__init__.py

@@ -12,7 +12,7 @@ supported services. Can check the output of local commands as well.
 
 from __future__ import annotations
 
-__min_python_version__ = (3, 9)  # minimum version of Python required to run; supported until 5 October 2023
+__min_python_version__ = (3, 10)  # minimum version of Python required to run; supported until fall 2025
 
 
 __project_name__ = __package__
@@ -22,7 +22,7 @@ __project_name__ = __package__
 # * MINOR version when you add functionality in a backwards compatible manner, and
 # * MICRO or PATCH version when you make backwards compatible bug fixes. We no longer use '0'
 # If unsure on increments, use pkg_resources.parse_version to parse
-__version__ = '3.24.1'
+__version__ = '3.26.0'
 __description__ = (
     'Check web (or command output) for changes since last run and notify.\n'
     '\n'
@@ -36,10 +36,8 @@ __code_url__ = f'https://github.com/mborsetti/{__project_name__}/'
 __docs_url__ = f'https://{__project_name__}.readthedocs.io/'
 __user_agent__ = f'{__project_name__}/{__version__} (+{__url__})'
 
-from typing import Union
 
-
-def init_data() -> dict[str, Union[str, tuple]]:
+def init_data() -> dict[str, str | tuple]:
     """Returns dict of globals (used in testing).
 
     :returns: dict of globals()

webchanges-3.26.0/webchanges/_vendored/headers.py

@@ -0,0 +1,313 @@
+"""
+Vendored version of httpx.Headers class from httpx v0.27.0 released on 21-Feb-24
+https://github.com/encode/httpx/releases/tag/0.27.0.
+
+Allows us to load this class in case httpx isn't installed.
+
+See https://github.com/psf/requests and https://github.com/encode/httpx/blob/master/httpx/_models.py
+"""
+
+from __future__ import annotations
+
+from typing import (
+    Any,
+    AnyStr,
+    ItemsView,
+    Iterable,
+    Iterator,
+    KeysView,
+    Mapping,
+    MutableMapping,
+    Sequence,
+    TypeAlias,
+    ValuesView,
+)
+
+HeaderTypes: TypeAlias = (
+    'Headers' | Mapping[str, str] | Mapping[bytes, bytes] | Sequence[tuple[str, str]] | Sequence[tuple[bytes, bytes]]
+)
+
+
+def normalize_header_key(
+    value: str | bytes,
+    lower: bool,
+    encoding: str | None = None,
+) -> bytes:
+    """
+    Coerce str/bytes into a strictly byte-wise HTTP header key.
+    """
+    if isinstance(value, bytes):
+        bytes_value = value
+    else:
+        bytes_value = value.encode(encoding or 'ascii')
+
+    return bytes_value.lower() if lower else bytes_value
+
+
+def normalize_header_value(value: str | bytes, encoding: str | None = None) -> bytes:
+    """
+    Coerce str/bytes into a strictly byte-wise HTTP header value.
+    """
+    if isinstance(value, bytes):
+        return value
+    return value.encode(encoding or 'ascii')
+
+
+SENSITIVE_HEADERS = {'authorization', 'proxy-authorization'}
+
+
+def obfuscate_sensitive_headers(
+    items: Iterable[tuple[AnyStr, AnyStr]],
+) -> Iterator[tuple[AnyStr, AnyStr]]:
+    for k, v in items:
+        if to_str(k.lower()) in SENSITIVE_HEADERS:
+            v = to_bytes_or_str('[secure]', match_type_of=v)
+        yield k, v
+
+
+def to_str(value: str | bytes, encoding: str = 'utf-8') -> str:
+    return value if isinstance(value, str) else value.decode(encoding)
+
+
+def to_bytes_or_str(value: str, match_type_of: AnyStr) -> AnyStr:
+    return value if isinstance(match_type_of, str) else value.encode()
+
+
+class Headers(MutableMapping[str, str]):
+    """
+    HTTP headers, as a case-insensitive multi-dict.
+    """
+
+    def __init__(
+        self,
+        headers: HeaderTypes | None = None,
+        encoding: str | None = None,
+    ) -> None:
+        if headers is None:
+            self._list: list[tuple[bytes, bytes, bytes]] = []
+        elif isinstance(headers, Headers):
+            self._list = list(headers._list)
+        elif isinstance(headers, Mapping):
+            self._list = [
+                (
+                    normalize_header_key(k, lower=False, encoding=encoding),
+                    normalize_header_key(k, lower=True, encoding=encoding),
+                    normalize_header_value(v, encoding),
+                )
+                for k, v in headers.items()
+            ]
+        else:
+            self._list = [
+                (
+                    normalize_header_key(k, lower=False, encoding=encoding),
+                    normalize_header_key(k, lower=True, encoding=encoding),
+                    normalize_header_value(v, encoding),
+                )
+                for k, v in headers
+            ]
+
+        self._encoding = encoding
+
+    @property
+    def encoding(self) -> str:
+        """
+        Header encoding is mandated as ascii, but we allow fallbacks to utf-8
+        or iso-8859-1.
+        """
+        if self._encoding is None:
+            for encoding in ['ascii', 'utf-8']:
+                for key, value in self.raw:
+                    try:
+                        key.decode(encoding)
+                        value.decode(encoding)
+                    except UnicodeDecodeError:
+                        break
+                else:
+                    # The else block runs if 'break' did not occur, meaning
+                    # all values fitted the encoding.
+                    self._encoding = encoding
+                    break
+            else:
+                # The ISO-8859-1 encoding covers all 256 code points in a byte,
+                # so will never raise decode errors.
+                self._encoding = 'iso-8859-1'
+        return self._encoding
+
+    @encoding.setter
+    def encoding(self, value: str) -> None:
+        self._encoding = value
+
+    @property
+    def raw(self) -> list[tuple[bytes, bytes]]:
+        """
+        Returns a list of the raw header items, as byte pairs.
+        """
+        return [(raw_key, value) for raw_key, _, value in self._list]
+
+    def keys(self) -> KeysView[str]:
+        return {key.decode(self.encoding): None for _, key, value in self._list}.keys()
+
+    def values(self) -> ValuesView[str]:
+        values_dict: dict[str, str] = {}
+        for _, key, value in self._list:
+            str_key = key.decode(self.encoding)
+            str_value = value.decode(self.encoding)
+            if str_key in values_dict:
+                values_dict[str_key] += f', {str_value}'
+            else:
+                values_dict[str_key] = str_value
+        return values_dict.values()
+
+    def items(self) -> ItemsView[str, str]:
+        """
+        Return `(key, value)` items of headers. Concatenate headers
+        into a single comma separated value when a key occurs multiple times.
+        """
+        values_dict: dict[str, str] = {}
+        for _, key, value in self._list:
+            str_key = key.decode(self.encoding)
+            str_value = value.decode(self.encoding)
+            if str_key in values_dict:
+                values_dict[str_key] += f', {str_value}'
+            else:
+                values_dict[str_key] = str_value
+        return values_dict.items()
+
+    def multi_items(self) -> list[tuple[str, str]]:
+        """
+        Return a list of `(key, value)` pairs of headers. Allow multiple
+        occurrences of the same key without concatenating into a single
+        comma separated value.
+        """
+        return [(key.decode(self.encoding), value.decode(self.encoding)) for _, key, value in self._list]
+
+    def get(self, key: str, default: Any = None) -> Any:
+        """
+        Return a header value. If multiple occurrences of the header occur
+        then concatenate them together with commas.
+        """
+        try:
+            return self[key]
+        except KeyError:
+            return default
+
+    def get_list(self, key: str, split_commas: bool = False) -> list[str]:
+        """
+        Return a list of all header values for a given key.
+        If `split_commas=True` is passed, then any comma separated header
+        values are split into multiple return strings.
+        """
+        get_header_key = key.lower().encode(self.encoding)
+
+        values = [
+            item_value.decode(self.encoding)
+            for _, item_key, item_value in self._list
+            if item_key.lower() == get_header_key
+        ]
+
+        if not split_commas:
+            return values
+
+        split_values = []
+        for value in values:
+            split_values.extend([item.strip() for item in value.split(',')])
+        return split_values
+
+    def update(self, headers: HeaderTypes | None = None) -> None:  # type: ignore[override]
+        headers = Headers(headers)
+        for key in headers.keys():
+            if key in self:
+                self.pop(key)
+        self._list.extend(headers._list)
+
+    def copy(self) -> Headers:
+        return Headers(self, encoding=self.encoding)
+
+    def __getitem__(self, key: str) -> str:
+        """
+        Return a single header value.
+
+        If there are multiple headers with the same key, then we concatenate
+        them with commas. See: https://tools.ietf.org/html/rfc7230#section-3.2.2
+        """
+        normalized_key = key.lower().encode(self.encoding)
+
+        items = [
+            header_value.decode(self.encoding)
+            for _, header_key, header_value in self._list
+            if header_key == normalized_key
+        ]
+
+        if items:
+            return ', '.join(items)
+
+        raise KeyError(key)
+
+    def __setitem__(self, key: str, value: str) -> None:
+        """
+        Set the header `key` to `value`, removing any duplicate entries.
+        Retains insertion order.
+        """
+        set_key = key.encode(self._encoding or 'utf-8')
+        set_value = value.encode(self._encoding or 'utf-8')
+        lookup_key = set_key.lower()
+
+        found_indexes = [idx for idx, (_, item_key, _) in enumerate(self._list) if item_key == lookup_key]
+
+        for idx in reversed(found_indexes[1:]):
+            del self._list[idx]
+
+        if found_indexes:
+            idx = found_indexes[0]
+            self._list[idx] = (set_key, lookup_key, set_value)
+        else:
+            self._list.append((set_key, lookup_key, set_value))
+
+    def __delitem__(self, key: str) -> None:
+        """
+        Remove the header `key`.
+        """
+        del_key = key.lower().encode(self.encoding)
+
+        pop_indexes = [idx for idx, (_, item_key, _) in enumerate(self._list) if item_key.lower() == del_key]
+
+        if not pop_indexes:
+            raise KeyError(key)
+
+        for idx in reversed(pop_indexes):
+            del self._list[idx]
+
+    def __contains__(self, key: Any) -> bool:
+        header_key = key.lower().encode(self.encoding)
+        return header_key in [key for _, key, _ in self._list]
+
+    def __iter__(self) -> Iterator[Any]:
+        return iter(self.keys())
+
+    def __len__(self) -> int:
+        return len(self._list)
+
+    def __eq__(self, other: Any) -> bool:
+        try:
+            other_headers = Headers(other)
+        except ValueError:
+            return False
+
+        self_list = [(key, value) for _, key, value in self._list]
+        other_list = [(key, value) for _, key, value in other_headers._list]
+        return sorted(self_list) == sorted(other_list)
+
+    def __repr__(self) -> str:
+        class_name = self.__class__.__name__
+
+        encoding_str = ''
+        if self.encoding != 'ascii':
+            encoding_str = f', encoding={self.encoding!r}'
+
+        as_list = list(obfuscate_sensitive_headers(self.multi_items()))
+        as_dict = dict(as_list)
+
+        no_duplicate_keys = len(as_dict) == len(as_list)
+        if no_duplicate_keys:
+            return f'{class_name}({as_dict!r}{encoding_str})'
+        return f'{class_name}({as_list!r}{encoding_str})'