webchanges 3.28.1__tar.gz → 3.29.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {webchanges-3.28.1/webchanges.egg-info → webchanges-3.29.0}/PKG-INFO +26 -15
- {webchanges-3.28.1 → webchanges-3.29.0}/README.rst +23 -13
- {webchanges-3.28.1 → webchanges-3.29.0}/webchanges/__init__.py +1 -1
- webchanges-3.29.0/webchanges/__main__.py +10 -0
- {webchanges-3.28.1 → webchanges-3.29.0}/webchanges/cli.py +11 -7
- {webchanges-3.28.1 → webchanges-3.29.0}/webchanges/command.py +2 -2
- {webchanges-3.28.1 → webchanges-3.29.0}/webchanges/differs.py +152 -83
- {webchanges-3.28.1 → webchanges-3.29.0}/webchanges/filters.py +5 -2
- {webchanges-3.28.1 → webchanges-3.29.0}/webchanges/handler.py +30 -28
- {webchanges-3.28.1 → webchanges-3.29.0}/webchanges/jobs.py +4 -7
- {webchanges-3.28.1 → webchanges-3.29.0}/webchanges/reporters.py +4 -4
- {webchanges-3.28.1 → webchanges-3.29.0}/webchanges/storage.py +2 -2
- {webchanges-3.28.1 → webchanges-3.29.0/webchanges.egg-info}/PKG-INFO +26 -15
- {webchanges-3.28.1 → webchanges-3.29.0}/webchanges.egg-info/SOURCES.txt +1 -0
- {webchanges-3.28.1 → webchanges-3.29.0}/LICENSE +0 -0
- {webchanges-3.28.1 → webchanges-3.29.0}/MANIFEST.in +0 -0
- {webchanges-3.28.1 → webchanges-3.29.0}/pyproject.toml +0 -0
- {webchanges-3.28.1 → webchanges-3.29.0}/requirements.txt +0 -0
- {webchanges-3.28.1 → webchanges-3.29.0}/setup.cfg +0 -0
- {webchanges-3.28.1 → webchanges-3.29.0}/webchanges/_vendored/__init__.py +0 -0
- {webchanges-3.28.1 → webchanges-3.29.0}/webchanges/_vendored/headers.py +0 -0
- {webchanges-3.28.1 → webchanges-3.29.0}/webchanges/_vendored/packaging_version.py +0 -0
- {webchanges-3.28.1 → webchanges-3.29.0}/webchanges/config.py +0 -0
- {webchanges-3.28.1 → webchanges-3.29.0}/webchanges/mailer.py +0 -0
- {webchanges-3.28.1 → webchanges-3.29.0}/webchanges/main.py +0 -0
- {webchanges-3.28.1 → webchanges-3.29.0}/webchanges/py.typed +0 -0
- {webchanges-3.28.1 → webchanges-3.29.0}/webchanges/storage_minidb.py +0 -0
- {webchanges-3.28.1 → webchanges-3.29.0}/webchanges/util.py +0 -0
- {webchanges-3.28.1 → webchanges-3.29.0}/webchanges/worker.py +0 -0
- {webchanges-3.28.1 → webchanges-3.29.0}/webchanges.egg-info/dependency_links.txt +0 -0
- {webchanges-3.28.1 → webchanges-3.29.0}/webchanges.egg-info/entry_points.txt +0 -0
- {webchanges-3.28.1 → webchanges-3.29.0}/webchanges.egg-info/requires.txt +0 -0
- {webchanges-3.28.1 → webchanges-3.29.0}/webchanges.egg-info/top_level.txt +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
|
-
Metadata-Version: 2.
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
2
|
Name: webchanges
|
|
3
|
-
Version: 3.
|
|
3
|
+
Version: 3.29.0
|
|
4
4
|
Summary: Web Changes Delivered. AI-Summarized. Totally Anonymous.
|
|
5
5
|
Author-email: Mike Borsetti <mike+webchanges@borsetti.com>
|
|
6
6
|
Maintainer-email: Mike Borsetti <mike+webchanges@borsetti.com>
|
|
@@ -163,6 +163,7 @@ Provides-Extra: safe-password
|
|
|
163
163
|
Requires-Dist: keyring; extra == "safe-password"
|
|
164
164
|
Provides-Extra: all
|
|
165
165
|
Requires-Dist: webchanges[beautify,bs4,deepdiff_xml,html5lib,ical2text,imagediff,jq,matrix,ocr,pdf2text,pushbullet,pushover,pypdf_crypto,redis,requests,safe_password,use_browser,xmpp]; extra == "all"
|
|
166
|
+
Dynamic: license-file
|
|
166
167
|
|
|
167
168
|
.. role:: underline
|
|
168
169
|
:class: underline
|
|
@@ -195,7 +196,9 @@ For Generative AI summaries (BETA), you need a free `API Key from Google Cloud A
|
|
|
195
196
|
|
|
196
197
|
Installation
|
|
197
198
|
============
|
|
198
|
-
|
|
199
|
+
|pypi_version| |format| |status| |security|
|
|
200
|
+
|
|
201
|
+
Install **webchanges** with:
|
|
199
202
|
|
|
200
203
|
.. code-block:: bash
|
|
201
204
|
|
|
@@ -203,7 +206,7 @@ Install **webchanges** |pypi_version| |format| |status| |security| with:
|
|
|
203
206
|
|
|
204
207
|
Running in Docker
|
|
205
208
|
-----------------
|
|
206
|
-
**webchanges** can easily run in a container
|
|
209
|
+
**webchanges** can easily run in a container and you will find a `Docker <https://www.docker.com/>`__ implementation
|
|
207
210
|
`here <https://github.com/yubiuser/webchanges-docker>`__.
|
|
208
211
|
|
|
209
212
|
|
|
@@ -247,20 +250,19 @@ Schedule
|
|
|
247
250
|
--------
|
|
248
251
|
**webchanges** leverages the power of a system scheduler:
|
|
249
252
|
|
|
250
|
-
- On Linux you can use cron,
|
|
251
|
-
|
|
252
|
-
used cron before);
|
|
253
|
+
- On Linux you can use cron, with the help of a tool like `crontab.guru <https://crontab.guru>`__ (help `here
|
|
254
|
+
<https://www.computerhope.com/unix/ucrontab.htm>`__);
|
|
253
255
|
- On Windows you can use `Windows Task Scheduler <https://en.wikipedia.org/wiki/Windows_Task_Scheduler>`__;
|
|
254
256
|
- On macOS you can use `launchd <https://developer.apple
|
|
255
|
-
.com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/ScheduledJobs.html>`__ (
|
|
256
|
-
<https://launchd.info/>`__
|
|
257
|
+
.com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/ScheduledJobs.html>`__ (help `here
|
|
258
|
+
<https://launchd.info/>`__).
|
|
257
259
|
|
|
258
260
|
|
|
259
261
|
Code
|
|
260
262
|
====
|
|
261
|
-
|coveralls| |issues|
|
|
263
|
+
|coveralls| |issues| |code_style|
|
|
262
264
|
|
|
263
|
-
The code
|
|
265
|
+
The code, issues tracker, and discussions are hosted on `GitHub <https://github.com/mborsetti/webchanges>`__.
|
|
264
266
|
|
|
265
267
|
|
|
266
268
|
Contributing
|
|
@@ -344,7 +346,7 @@ Example enhancements to HTML reporting:
|
|
|
344
346
|
.. |format| image:: https://img.shields.io/pypi/format/webchanges.svg
|
|
345
347
|
:target: https://pypi.org/project/webchanges/
|
|
346
348
|
:alt: Kit format
|
|
347
|
-
.. |downloads| image:: https://
|
|
349
|
+
.. |downloads| image:: https://img.shields.io/pypi/dm/webchanges.svg
|
|
348
350
|
:target: https://www.pepy.tech/project/webchanges
|
|
349
351
|
:alt: PyPI downloads
|
|
350
352
|
.. |license| image:: https://img.shields.io/pypi/l/webchanges.svg
|
|
@@ -356,15 +358,24 @@ Example enhancements to HTML reporting:
|
|
|
356
358
|
.. |readthedocs| image:: https://img.shields.io/readthedocs/webchanges/stable.svg?label=
|
|
357
359
|
:target: https://webchanges.readthedocs.io/
|
|
358
360
|
:alt: Documentation status
|
|
359
|
-
.. |
|
|
361
|
+
.. |old_CI| image:: https://github.com/mborsetti/webchanges/actions/workflows/ci-cd.yaml/badge.svg?event=push
|
|
362
|
+
:target: https://github.com/mborsetti/webchanges/actions
|
|
363
|
+
:alt: CI testing status
|
|
364
|
+
.. |CI| image:: https://img.shields.io/github/check-runs/mborsetti/webchanges/main
|
|
360
365
|
:target: https://github.com/mborsetti/webchanges/actions
|
|
361
366
|
:alt: CI testing status
|
|
362
|
-
.. |
|
|
367
|
+
.. |old_coveralls| image:: https://coveralls.io/repos/github/mborsetti/webchanges/badge.svg?branch=main
|
|
368
|
+
:target: https://coveralls.io/github/mborsetti/webchanges?branch=main
|
|
369
|
+
:alt: Code coverage by Coveralls
|
|
370
|
+
.. |coveralls| image:: https://img.shields.io/coverallsCoverage/github/mborsetti/webchanges.svg
|
|
363
371
|
:target: https://coveralls.io/github/mborsetti/webchanges?branch=main
|
|
364
372
|
:alt: Code coverage by Coveralls
|
|
373
|
+
.. |code_style| image:: https://img.shields.io/badge/code%20style-black-000000.svg
|
|
374
|
+
:target: https://github.com/psf/black
|
|
375
|
+
:alt: Code style black
|
|
365
376
|
.. |status| image:: https://img.shields.io/pypi/status/webchanges.svg
|
|
366
377
|
:target: https://pypi.org/project/webchanges/
|
|
367
378
|
:alt: Package stability
|
|
368
|
-
.. |security| image:: https://img.shields.io/badge/security-bandit-
|
|
379
|
+
.. |security| image:: https://img.shields.io/badge/security-bandit-green.svg
|
|
369
380
|
:target: https://github.com/PyCQA/bandit
|
|
370
381
|
:alt: Security Status
|
|
@@ -29,7 +29,9 @@ For Generative AI summaries (BETA), you need a free `API Key from Google Cloud A
|
|
|
29
29
|
|
|
30
30
|
Installation
|
|
31
31
|
============
|
|
32
|
-
|
|
32
|
+
|pypi_version| |format| |status| |security|
|
|
33
|
+
|
|
34
|
+
Install **webchanges** with:
|
|
33
35
|
|
|
34
36
|
.. code-block:: bash
|
|
35
37
|
|
|
@@ -37,7 +39,7 @@ Install **webchanges** |pypi_version| |format| |status| |security| with:
|
|
|
37
39
|
|
|
38
40
|
Running in Docker
|
|
39
41
|
-----------------
|
|
40
|
-
**webchanges** can easily run in a container
|
|
42
|
+
**webchanges** can easily run in a container and you will find a `Docker <https://www.docker.com/>`__ implementation
|
|
41
43
|
`here <https://github.com/yubiuser/webchanges-docker>`__.
|
|
42
44
|
|
|
43
45
|
|
|
@@ -81,20 +83,19 @@ Schedule
|
|
|
81
83
|
--------
|
|
82
84
|
**webchanges** leverages the power of a system scheduler:
|
|
83
85
|
|
|
84
|
-
- On Linux you can use cron,
|
|
85
|
-
|
|
86
|
-
used cron before);
|
|
86
|
+
- On Linux you can use cron, with the help of a tool like `crontab.guru <https://crontab.guru>`__ (help `here
|
|
87
|
+
<https://www.computerhope.com/unix/ucrontab.htm>`__);
|
|
87
88
|
- On Windows you can use `Windows Task Scheduler <https://en.wikipedia.org/wiki/Windows_Task_Scheduler>`__;
|
|
88
89
|
- On macOS you can use `launchd <https://developer.apple
|
|
89
|
-
.com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/ScheduledJobs.html>`__ (
|
|
90
|
-
<https://launchd.info/>`__
|
|
90
|
+
.com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/ScheduledJobs.html>`__ (help `here
|
|
91
|
+
<https://launchd.info/>`__).
|
|
91
92
|
|
|
92
93
|
|
|
93
94
|
Code
|
|
94
95
|
====
|
|
95
|
-
|coveralls| |issues|
|
|
96
|
+
|coveralls| |issues| |code_style|
|
|
96
97
|
|
|
97
|
-
The code
|
|
98
|
+
The code, issues tracker, and discussions are hosted on `GitHub <https://github.com/mborsetti/webchanges>`__.
|
|
98
99
|
|
|
99
100
|
|
|
100
101
|
Contributing
|
|
@@ -178,7 +179,7 @@ Example enhancements to HTML reporting:
|
|
|
178
179
|
.. |format| image:: https://img.shields.io/pypi/format/webchanges.svg
|
|
179
180
|
:target: https://pypi.org/project/webchanges/
|
|
180
181
|
:alt: Kit format
|
|
181
|
-
.. |downloads| image:: https://
|
|
182
|
+
.. |downloads| image:: https://img.shields.io/pypi/dm/webchanges.svg
|
|
182
183
|
:target: https://www.pepy.tech/project/webchanges
|
|
183
184
|
:alt: PyPI downloads
|
|
184
185
|
.. |license| image:: https://img.shields.io/pypi/l/webchanges.svg
|
|
@@ -190,15 +191,24 @@ Example enhancements to HTML reporting:
|
|
|
190
191
|
.. |readthedocs| image:: https://img.shields.io/readthedocs/webchanges/stable.svg?label=
|
|
191
192
|
:target: https://webchanges.readthedocs.io/
|
|
192
193
|
:alt: Documentation status
|
|
193
|
-
.. |
|
|
194
|
+
.. |old_CI| image:: https://github.com/mborsetti/webchanges/actions/workflows/ci-cd.yaml/badge.svg?event=push
|
|
195
|
+
:target: https://github.com/mborsetti/webchanges/actions
|
|
196
|
+
:alt: CI testing status
|
|
197
|
+
.. |CI| image:: https://img.shields.io/github/check-runs/mborsetti/webchanges/main
|
|
194
198
|
:target: https://github.com/mborsetti/webchanges/actions
|
|
195
199
|
:alt: CI testing status
|
|
196
|
-
.. |
|
|
200
|
+
.. |old_coveralls| image:: https://coveralls.io/repos/github/mborsetti/webchanges/badge.svg?branch=main
|
|
201
|
+
:target: https://coveralls.io/github/mborsetti/webchanges?branch=main
|
|
202
|
+
:alt: Code coverage by Coveralls
|
|
203
|
+
.. |coveralls| image:: https://img.shields.io/coverallsCoverage/github/mborsetti/webchanges.svg
|
|
197
204
|
:target: https://coveralls.io/github/mborsetti/webchanges?branch=main
|
|
198
205
|
:alt: Code coverage by Coveralls
|
|
206
|
+
.. |code_style| image:: https://img.shields.io/badge/code%20style-black-000000.svg
|
|
207
|
+
:target: https://github.com/psf/black
|
|
208
|
+
:alt: Code style black
|
|
199
209
|
.. |status| image:: https://img.shields.io/pypi/status/webchanges.svg
|
|
200
210
|
:target: https://pypi.org/project/webchanges/
|
|
201
211
|
:alt: Package stability
|
|
202
|
-
.. |security| image:: https://img.shields.io/badge/security-bandit-
|
|
212
|
+
.. |security| image:: https://img.shields.io/badge/security-bandit-green.svg
|
|
203
213
|
:target: https://github.com/PyCQA/bandit
|
|
204
214
|
:alt: Security Status
|
|
@@ -22,7 +22,7 @@ __project_name__ = __package__
|
|
|
22
22
|
# * MINOR version when you add functionality in a backwards compatible manner, and
|
|
23
23
|
# * MICRO or PATCH version when you make backwards compatible bug fixes. We no longer use '0'
|
|
24
24
|
# If unsure on increments, use pkg_resources.parse_version to parse
|
|
25
|
-
__version__ = '3.
|
|
25
|
+
__version__ = '3.29.0'
|
|
26
26
|
__description__ = (
|
|
27
27
|
'Check web (or command output) for changes since last run and notify.\n'
|
|
28
28
|
'\n'
|
|
@@ -244,19 +244,18 @@ def first_run(command_config: CommandConfig) -> None:
|
|
|
244
244
|
def load_hooks(hooks_file: Path) -> None:
|
|
245
245
|
"""Load hooks file."""
|
|
246
246
|
if not hooks_file.is_file():
|
|
247
|
-
|
|
248
|
-
|
|
249
|
-
ImportWarning,
|
|
250
|
-
)
|
|
247
|
+
# do not use ImportWarning as it could be suppressed
|
|
248
|
+
warnings.warn(f'Hooks file not imported because {hooks_file} is not a file', RuntimeWarning)
|
|
251
249
|
return
|
|
252
250
|
|
|
253
251
|
hooks_file_errors = file_ownership_checks(hooks_file)
|
|
254
252
|
if hooks_file_errors:
|
|
253
|
+
logger.debug('Here should come the warning')
|
|
254
|
+
# do not use ImportWarning as it could be suppressed
|
|
255
255
|
warnings.warn(
|
|
256
|
-
f
|
|
257
|
-
f" {' and '.join(hooks_file_errors)}.\n"
|
|
256
|
+
f"Hooks file {hooks_file} not not imported because{' and '.join(hooks_file_errors)}.\n"
|
|
258
257
|
f'(see {__docs_url__}en/stable/hooks.html#important-note-for-hooks-file)',
|
|
259
|
-
|
|
258
|
+
RuntimeWarning,
|
|
260
259
|
)
|
|
261
260
|
else:
|
|
262
261
|
logger.info(f'Importing hooks module from {hooks_file}')
|
|
@@ -372,6 +371,10 @@ def main() -> None: # pragma: no cover
|
|
|
372
371
|
# Set up the logger to verbose if needed
|
|
373
372
|
setup_logger(command_config.verbose, command_config.log_file)
|
|
374
373
|
|
|
374
|
+
# log defaults
|
|
375
|
+
logger.debug(f'Default config path is {config_path}')
|
|
376
|
+
logger.debug(f'Default data path is {data_path}')
|
|
377
|
+
|
|
375
378
|
# For speed, run these here
|
|
376
379
|
handle_unitialized_actions(command_config)
|
|
377
380
|
|
|
@@ -401,6 +404,7 @@ def main() -> None: # pragma: no cover
|
|
|
401
404
|
|
|
402
405
|
# load config (which for syntax checking requires hooks to be loaded too)
|
|
403
406
|
if command_config.hooks_files:
|
|
407
|
+
logger.debug(f'Hooks files to be loaded: {command_config.hooks_files}')
|
|
404
408
|
for hooks_file in command_config.hooks_files:
|
|
405
409
|
load_hooks(hooks_file)
|
|
406
410
|
config_storage.load()
|
|
@@ -498,8 +498,8 @@ class UrlwatchCommand:
|
|
|
498
498
|
|
|
499
499
|
def test_differ(self, arg_test_differ: list[str]) -> int:
|
|
500
500
|
"""
|
|
501
|
-
Runs diffs for a job on all the saved snapshots and outputs the result to stdout or
|
|
502
|
-
|
|
501
|
+
Runs diffs for a job on all the saved snapshots and outputs the result to stdout or the reporter selected
|
|
502
|
+
with --test-reporter.
|
|
503
503
|
|
|
504
504
|
:param arg_test_differ: Either the job_id or a list containing [job_id, max_diffs]
|
|
505
505
|
:return: 1 if error, 0 if successful.
|
|
@@ -151,12 +151,13 @@ class DifferBase(metaclass=TrackSubClasses):
|
|
|
151
151
|
) -> dict[str, Any]:
|
|
152
152
|
"""Obtain differ subdirectives that also contains defaults from the configuration.
|
|
153
153
|
|
|
154
|
-
:param
|
|
154
|
+
:param differ_spec: The differ as entered by the user; use "unified" if empty.
|
|
155
155
|
:param directives: The differ directives as stated in the job.
|
|
156
|
+
:param config: The configuration.
|
|
156
157
|
:returns: directives inclusive of configuration defaults.
|
|
157
158
|
"""
|
|
158
159
|
if config is None:
|
|
159
|
-
logger.
|
|
160
|
+
logger.info('No configuration object found to look for differ defaults')
|
|
160
161
|
return directives
|
|
161
162
|
cfg = config.get('differ_defaults')
|
|
162
163
|
if isinstance(cfg, dict):
|
|
@@ -261,9 +262,13 @@ class DifferBase(metaclass=TrackSubClasses):
|
|
|
261
262
|
_unfiltered_diff: dict[Literal['text', 'markdown', 'html'], str] | None = None,
|
|
262
263
|
tz: ZoneInfo | None = None,
|
|
263
264
|
) -> dict[Literal['text', 'markdown', 'html'], str]:
|
|
264
|
-
"""
|
|
265
|
-
|
|
266
|
-
|
|
265
|
+
"""Generate a formatted diff representation of data changes.
|
|
266
|
+
|
|
267
|
+
Creates a diff representation in one or more output formats (text, markdown, or HTML).
|
|
268
|
+
At minimum, this function must return output in the format specified by 'report_kind'.
|
|
269
|
+
As results are memoized for performance optimization, it can generate up to all three formats simultaneously.
|
|
270
|
+
|
|
271
|
+
:param state: The JobState.
|
|
267
272
|
|
|
268
273
|
:param directives: The directives.
|
|
269
274
|
:param report_kind: The report_kind for which a diff must be generated (at a minimum).
|
|
@@ -281,12 +286,14 @@ class DifferBase(metaclass=TrackSubClasses):
|
|
|
281
286
|
timestamp: float,
|
|
282
287
|
tz: ZoneInfo | None = None,
|
|
283
288
|
) -> str:
|
|
284
|
-
"""
|
|
285
|
-
|
|
289
|
+
"""Format a timestamp as an RFC 5322 compliant datetime string.
|
|
290
|
+
|
|
291
|
+
Converts a numeric timestamp to a formatted datetime string following the RFC 5322 (email) standard. When a
|
|
292
|
+
timezone is provided, its full name, if known, is appended.
|
|
286
293
|
|
|
287
294
|
:param timestamp: The timestamp.
|
|
288
295
|
:param tz: The IANA timezone of the report.
|
|
289
|
-
:returns: A datetime string in RFC 5322 (email) format.
|
|
296
|
+
:returns: A datetime string in RFC 5322 (email) format or 'NEW' if timestamp is 0.
|
|
290
297
|
"""
|
|
291
298
|
if timestamp:
|
|
292
299
|
dt = datetime.fromtimestamp(timestamp).astimezone(tz=tz)
|
|
@@ -549,6 +556,7 @@ class CommandDiffer(DifferBase):
|
|
|
549
556
|
|
|
550
557
|
__supported_directives__ = {
|
|
551
558
|
'command': 'The command to execute',
|
|
559
|
+
'is_html': 'Whether the output of the command is HTML',
|
|
552
560
|
}
|
|
553
561
|
|
|
554
562
|
re_ptags = re.compile(r'^<p>|</p>$')
|
|
@@ -562,15 +570,30 @@ class CommandDiffer(DifferBase):
|
|
|
562
570
|
_unfiltered_diff: dict[Literal['text', 'markdown', 'html'], str] | None = None,
|
|
563
571
|
tz: ZoneInfo | None = None,
|
|
564
572
|
) -> dict[Literal['text', 'markdown', 'html'], str]:
|
|
573
|
+
if self.job.monospace:
|
|
574
|
+
head_html = '\n'.join(
|
|
575
|
+
[
|
|
576
|
+
'<span style="font-family:monospace;white-space:pre-wrap;">',
|
|
577
|
+
# f"Using command differ: {directives['command']}",
|
|
578
|
+
f'<span style="color:darkred;">--- @ {self.make_timestamp(self.state.old_timestamp, tz)}</span>',
|
|
579
|
+
f'<span style="color:darkgreen;">+++ @ {self.make_timestamp(self.state.new_timestamp, tz)}</span>',
|
|
580
|
+
]
|
|
581
|
+
)
|
|
582
|
+
else:
|
|
583
|
+
head_html = '<br>\n'.join(
|
|
584
|
+
[
|
|
585
|
+
'<span style="font-family:monospace;">',
|
|
586
|
+
# f"Using command differ: {directives['command']}",
|
|
587
|
+
f'<span style="color:darkred;">--- @ {self.make_timestamp(self.state.old_timestamp, tz)}</span>',
|
|
588
|
+
f'<span style="color:darkgreen;">+++ @ {self.make_timestamp(self.state.new_timestamp, tz)}</span>',
|
|
589
|
+
'</span>',
|
|
590
|
+
]
|
|
591
|
+
)
|
|
592
|
+
|
|
565
593
|
out_diff: dict[Literal['text', 'markdown', 'html'], str] = {}
|
|
566
594
|
command = directives['command']
|
|
567
|
-
if
|
|
568
|
-
|
|
569
|
-
and not command.startswith('wdiff')
|
|
570
|
-
and _unfiltered_diff is not None
|
|
571
|
-
and 'text' in _unfiltered_diff
|
|
572
|
-
):
|
|
573
|
-
diff = _unfiltered_diff['text']
|
|
595
|
+
if report_kind == 'html' and _unfiltered_diff is not None and 'text' in _unfiltered_diff:
|
|
596
|
+
diff_text = ''.join(_unfiltered_diff['text'].splitlines(keepends=True)[2:])
|
|
574
597
|
else:
|
|
575
598
|
old_data = self.state.old_data
|
|
576
599
|
new_data = self.state.new_data
|
|
@@ -606,12 +629,17 @@ class CommandDiffer(DifferBase):
|
|
|
606
629
|
) from subprocess.CalledProcessError(proc.returncode, cmdline)
|
|
607
630
|
if proc.returncode == 0:
|
|
608
631
|
self.state.verb = 'changed,no_report'
|
|
632
|
+
logger.info(
|
|
633
|
+
f"Job {self.job.index_number}: Command in differ 'command' returned 0 (no report) "
|
|
634
|
+
f'({self.job.get_location()})'
|
|
635
|
+
)
|
|
609
636
|
return {'text': '', 'markdown': '', 'html': ''}
|
|
610
|
-
|
|
637
|
+
head_text = '\n'.join(
|
|
611
638
|
[
|
|
612
|
-
f
|
|
613
|
-
f'--- @ {self.make_timestamp(self.state.old_timestamp, tz)}',
|
|
614
|
-
f'+++ @ {self.make_timestamp(self.state.new_timestamp, tz)}',
|
|
639
|
+
# f"Using command differ: {directives['command']}",
|
|
640
|
+
f'\033[91m--- @ {self.make_timestamp(self.state.old_timestamp, tz)}\033[0m',
|
|
641
|
+
f'\033[92m+++ @ {self.make_timestamp(self.state.new_timestamp, tz)}\033[0m',
|
|
642
|
+
'',
|
|
615
643
|
]
|
|
616
644
|
)
|
|
617
645
|
diff = proc.stdout
|
|
@@ -625,20 +653,33 @@ class CommandDiffer(DifferBase):
|
|
|
625
653
|
if any(x in line for x in {'{+', '+}', '[-', '-]'}):
|
|
626
654
|
keeplines.append(line)
|
|
627
655
|
diff = ''.join(keeplines)
|
|
628
|
-
|
|
629
|
-
|
|
630
|
-
|
|
631
|
-
|
|
632
|
-
|
|
633
|
-
|
|
634
|
-
|
|
656
|
+
if directives.get('is_html'):
|
|
657
|
+
diff_text = self.html2text(diff)
|
|
658
|
+
out_diff.update(
|
|
659
|
+
{
|
|
660
|
+
'text': head_text + diff_text,
|
|
661
|
+
'markdown': head_text + diff_text,
|
|
662
|
+
'html': head_html + diff,
|
|
663
|
+
}
|
|
664
|
+
)
|
|
665
|
+
else:
|
|
666
|
+
diff_text = diff
|
|
667
|
+
out_diff.update(
|
|
668
|
+
{
|
|
669
|
+
'text': head_text + diff_text,
|
|
670
|
+
'markdown': head_text + diff_text,
|
|
671
|
+
}
|
|
672
|
+
)
|
|
635
673
|
|
|
636
|
-
if report_kind == 'html':
|
|
674
|
+
if report_kind == 'html' and 'html' not in out_diff:
|
|
637
675
|
if command.startswith('wdiff'):
|
|
638
676
|
# colorize output of wdiff
|
|
639
|
-
out_diff['html'] = self.wdiff_to_html(
|
|
677
|
+
out_diff['html'] = head_html + self.wdiff_to_html(diff_text)
|
|
640
678
|
else:
|
|
641
|
-
out_diff['html'] = html.escape(
|
|
679
|
+
out_diff['html'] = head_html + html.escape(diff_text)
|
|
680
|
+
|
|
681
|
+
if self.job.monospace and 'html' in out_diff:
|
|
682
|
+
out_diff['html'] += '</span>'
|
|
642
683
|
|
|
643
684
|
return out_diff
|
|
644
685
|
|
|
@@ -991,6 +1032,7 @@ class ImageDiffer(DifferBase):
|
|
|
991
1032
|
def ai_google(
|
|
992
1033
|
old_image: Image.Image,
|
|
993
1034
|
new_image: Image.Image,
|
|
1035
|
+
diff_image: Image.Image,
|
|
994
1036
|
directives: AiGoogleDirectives,
|
|
995
1037
|
) -> str:
|
|
996
1038
|
"""Summarize changes in image using Generative AI (ALPHA)."""
|
|
@@ -1073,10 +1115,6 @@ class ImageDiffer(DifferBase):
|
|
|
1073
1115
|
}
|
|
1074
1116
|
}
|
|
1075
1117
|
|
|
1076
|
-
# Create a diff image
|
|
1077
|
-
# diff_image = ImageChops.difference(old_image, new_image)
|
|
1078
|
-
# diff_image.format = new_image.format
|
|
1079
|
-
|
|
1080
1118
|
# upload to Google
|
|
1081
1119
|
additional_parts: list[dict[str, dict[str, str]]] = []
|
|
1082
1120
|
executor = ThreadPoolExecutor()
|
|
@@ -1085,7 +1123,7 @@ class ImageDiffer(DifferBase):
|
|
|
1085
1123
|
(
|
|
1086
1124
|
('old image', old_image),
|
|
1087
1125
|
('new image', new_image),
|
|
1088
|
-
|
|
1126
|
+
('differences image', diff_image),
|
|
1089
1127
|
),
|
|
1090
1128
|
):
|
|
1091
1129
|
if 'error' not in additional_part:
|
|
@@ -1107,22 +1145,33 @@ class ImageDiffer(DifferBase):
|
|
|
1107
1145
|
'focus on the most significant changes.'
|
|
1108
1146
|
)
|
|
1109
1147
|
model_prompt = (
|
|
1148
|
+
'You are a skilled visual analyst tasked with analyzing two versions of an image and summarizing the '
|
|
1149
|
+
'key differences between them. The audience for your summary is already familiar with the '
|
|
1150
|
+
"image's content, so you should focus only on the most significant differences.\n\n"
|
|
1110
1151
|
'**Instructions:**\n\n'
|
|
1111
|
-
|
|
1112
|
-
f"{additional_parts[
|
|
1113
|
-
|
|
1114
|
-
|
|
1115
|
-
'
|
|
1116
|
-
'
|
|
1117
|
-
'
|
|
1152
|
+
'1. Carefully examine the yellow areas in the image '
|
|
1153
|
+
f"{additional_parts[2]['file_data']['file_uri']}, identify the differences, and describe them.\n"
|
|
1154
|
+
f"2. Refer to the old version of the image {additional_parts[0]['file_data']['file_uri']} and the new "
|
|
1155
|
+
f" version {additional_parts[1]['file_data']['file_uri']}.\n"
|
|
1156
|
+
'3. You are only interested in those differences, such as additions, removals, or alterations, that '
|
|
1157
|
+
'modify the intended message or interpretation.\n'
|
|
1158
|
+
'4. Summarize the identified differences, except those ignored, in a clear and concise manner, '
|
|
1159
|
+
'explaining how the meaning has shifted or evolved in the new version compared to the old version only '
|
|
1160
|
+
'when necessary. Be specific and provide examples to illustrate your points when needed.\n'
|
|
1161
|
+
'5. If there are only additions to the image, then summarize the additions.\n'
|
|
1162
|
+
'6. Use Markdown formatting to structure your summary effectively. Use headings, bullet points, '
|
|
1163
|
+
'and other Markdown elements as needed to enhance readability.\n'
|
|
1164
|
+
'7. Restrict your analysis and summary to the information provided within these images. Do '
|
|
1165
|
+
'not introduce external information or assumptions.\n'
|
|
1118
1166
|
)
|
|
1119
|
-
summary = AIGoogleDiffer._send_to_model(
|
|
1167
|
+
summary, _ = AIGoogleDiffer._send_to_model(
|
|
1120
1168
|
self.job,
|
|
1121
1169
|
system_instructions,
|
|
1122
1170
|
model_prompt,
|
|
1123
1171
|
additional_parts=additional_parts, # type: ignore[arg-type]
|
|
1124
1172
|
directives=directives,
|
|
1125
1173
|
)
|
|
1174
|
+
|
|
1126
1175
|
return summary
|
|
1127
1176
|
|
|
1128
1177
|
data_type = directives.get('data_type', 'url')
|
|
@@ -1134,8 +1183,8 @@ class ImageDiffer(DifferBase):
|
|
|
1134
1183
|
if data_type == 'url':
|
|
1135
1184
|
old_image = load_image_from_web(self.state.old_data)
|
|
1136
1185
|
new_image = load_image_from_web(self.state.new_data)
|
|
1137
|
-
old_data = f' (<a href="{self.state.old_data}">Old image</a>)'
|
|
1138
|
-
new_data = f' (<a href="{self.state.new_data}">New image</a>)'
|
|
1186
|
+
old_data = f' (<a href="{self.state.old_data}" target="_blank">Old image</a>)'
|
|
1187
|
+
new_data = f' (<a href="{self.state.new_data}" target="_blank">New image</a>)'
|
|
1139
1188
|
elif data_type == 'ascii85':
|
|
1140
1189
|
old_image = load_image_from_ascii85(self.state.old_data)
|
|
1141
1190
|
new_image = load_image_from_ascii85(self.state.new_data)
|
|
@@ -1149,8 +1198,8 @@ class ImageDiffer(DifferBase):
|
|
|
1149
1198
|
else: # 'filename'
|
|
1150
1199
|
old_image = load_image_from_file(self.state.old_data)
|
|
1151
1200
|
new_image = load_image_from_file(self.state.new_data)
|
|
1152
|
-
old_data = f' (<a href="file://{self.state.old_data}">Old image</a>)'
|
|
1153
|
-
new_data = f' (<a href="file://{self.state.new_data}">New image</a>)'
|
|
1201
|
+
old_data = f' (<a href="file://{self.state.old_data}" target="_blank">Old image</a>)'
|
|
1202
|
+
new_data = f' (<a href="file://{self.state.new_data}" target="_blank">New image</a>)'
|
|
1154
1203
|
|
|
1155
1204
|
# Check formats TODO: is it needed? under which circumstances?
|
|
1156
1205
|
# if new_image.format != old_image.format:
|
|
@@ -1202,7 +1251,7 @@ class ImageDiffer(DifferBase):
|
|
|
1202
1251
|
# prepare AI summary
|
|
1203
1252
|
summary = ''
|
|
1204
1253
|
if 'ai_google' in directives:
|
|
1205
|
-
summary = ai_google(old_image, new_image, directives.get('ai_google', {}))
|
|
1254
|
+
summary = ai_google(old_image, new_image, diff_image, directives.get('ai_google', {}))
|
|
1206
1255
|
|
|
1207
1256
|
# Prepare HTML output
|
|
1208
1257
|
htm = [
|
|
@@ -1246,8 +1295,14 @@ class ImageDiffer(DifferBase):
|
|
|
1246
1295
|
)
|
|
1247
1296
|
footer = f'Summary generated by Google Generative AI (ai_google directive(s): {directives_text})'
|
|
1248
1297
|
return {
|
|
1249
|
-
'text':
|
|
1250
|
-
|
|
1298
|
+
'text': (
|
|
1299
|
+
f'{summary}\n\n\nA visualization of differences is available in {__package__} HTML reports.'
|
|
1300
|
+
f'\n------------\n{footer}'
|
|
1301
|
+
),
|
|
1302
|
+
'markdown': (
|
|
1303
|
+
f'{summary}\n\n\nA visualization of differences is available in {__package__} HTML reports.'
|
|
1304
|
+
f'\n* * *\n{footer}'
|
|
1305
|
+
),
|
|
1251
1306
|
'html': '<br>\n'.join(
|
|
1252
1307
|
[
|
|
1253
1308
|
mark_to_html(summary, extras={'tables'}).replace('<h2>', '<h3>').replace('</h2>', '</h3>'),
|
|
@@ -1298,20 +1353,17 @@ class AIGoogleDiffer(DifferBase):
|
|
|
1298
1353
|
model_prompt: str,
|
|
1299
1354
|
additional_parts: list[dict[str, str | dict[str, str]]] | None = None,
|
|
1300
1355
|
directives: AiGoogleDirectives | None = None,
|
|
1301
|
-
) -> str:
|
|
1302
|
-
"""Creates the summary request to the model"""
|
|
1356
|
+
) -> tuple[str, str]:
|
|
1357
|
+
"""Creates the summary request to the model; returns the summary and the version of the actual model used."""
|
|
1303
1358
|
api_version = '1beta'
|
|
1304
1359
|
if directives is None:
|
|
1305
1360
|
directives = {}
|
|
1306
|
-
|
|
1307
|
-
directives['model'] = 'gemini-2.0-flash' # also for footer
|
|
1308
|
-
model = directives.get('model')
|
|
1361
|
+
model = directives.get('model', 'gemini-2.0-flash')
|
|
1309
1362
|
timeout = directives.get('timeout', 300)
|
|
1310
1363
|
max_output_tokens = directives.get('max_output_tokens')
|
|
1311
1364
|
temperature = directives.get('temperature', 0.0)
|
|
1312
|
-
top_p = directives.get('top_p')
|
|
1365
|
+
top_p = directives.get('top_p', 1.0 if temperature == 0.0 else None)
|
|
1313
1366
|
top_k = directives.get('top_k')
|
|
1314
|
-
|
|
1315
1367
|
GOOGLE_AI_API_KEY = os.environ.get('GOOGLE_AI_API_KEY', '').rstrip()
|
|
1316
1368
|
if len(GOOGLE_AI_API_KEY) != 39:
|
|
1317
1369
|
logger.error(
|
|
@@ -1321,7 +1373,8 @@ class AIGoogleDiffer(DifferBase):
|
|
|
1321
1373
|
return (
|
|
1322
1374
|
f'## ERROR in summarizing changes using Google AI:\n'
|
|
1323
1375
|
f'Environment variable GOOGLE_AI_API_KEY not found or is of the incorrect length '
|
|
1324
|
-
f'{len(GOOGLE_AI_API_KEY)}.\n'
|
|
1376
|
+
f'{len(GOOGLE_AI_API_KEY)}.\n',
|
|
1377
|
+
'',
|
|
1325
1378
|
)
|
|
1326
1379
|
|
|
1327
1380
|
data: dict[str, Any] = {
|
|
@@ -1339,6 +1392,7 @@ class AIGoogleDiffer(DifferBase):
|
|
|
1339
1392
|
if directives.get('tools'):
|
|
1340
1393
|
data['tools'] = directives['tools']
|
|
1341
1394
|
logger.info(f'Job {job.index_number}: Making the content generation request to Google AI model {model}')
|
|
1395
|
+
model_version = model # default
|
|
1342
1396
|
try:
|
|
1343
1397
|
r = httpx.Client(http2=True).post( # noqa: S113 Call to httpx without timeout
|
|
1344
1398
|
f'https://generativelanguage.googleapis.com/v{api_version}/models/{model}:generateContent?'
|
|
@@ -1358,6 +1412,8 @@ class AIGoogleDiffer(DifferBase):
|
|
|
1358
1412
|
f'AI summary unavailable: Model did not return any candidate output:\n'
|
|
1359
1413
|
f'{jsonlib.dumps(result, ensure_ascii=True, indent=2)}'
|
|
1360
1414
|
)
|
|
1415
|
+
model_version = result['modelVersion']
|
|
1416
|
+
|
|
1361
1417
|
elif r.status_code == 400:
|
|
1362
1418
|
summary = (
|
|
1363
1419
|
f'AI summary unavailable: Received error from {r.url.host}: '
|
|
@@ -1375,7 +1431,7 @@ class AIGoogleDiffer(DifferBase):
|
|
|
1375
1431
|
f'AI summary unavailable: HTTP client error: {e} when requesting data from ' f'{e.request.url.host}'
|
|
1376
1432
|
)
|
|
1377
1433
|
|
|
1378
|
-
return summary
|
|
1434
|
+
return summary, model_version
|
|
1379
1435
|
|
|
1380
1436
|
def differ(
|
|
1381
1437
|
self,
|
|
@@ -1392,8 +1448,8 @@ class AIGoogleDiffer(DifferBase):
|
|
|
1392
1448
|
RuntimeWarning,
|
|
1393
1449
|
)
|
|
1394
1450
|
|
|
1395
|
-
def get_ai_summary(prompt: str, system_instructions: str) -> str:
|
|
1396
|
-
"""Generate AI summary from unified diff, or an error message"""
|
|
1451
|
+
def get_ai_summary(prompt: str, system_instructions: str) -> tuple[str, str]:
|
|
1452
|
+
"""Generate AI summary from unified diff, or an error message, plus the model version."""
|
|
1397
1453
|
GOOGLE_AI_API_KEY = os.environ.get('GOOGLE_AI_API_KEY', '').rstrip()
|
|
1398
1454
|
if len(GOOGLE_AI_API_KEY) != 39:
|
|
1399
1455
|
logger.error(
|
|
@@ -1403,12 +1459,10 @@ class AIGoogleDiffer(DifferBase):
|
|
|
1403
1459
|
return (
|
|
1404
1460
|
f'## ERROR in summarizing changes using {self.__kind__}:\n'
|
|
1405
1461
|
f'Environment variable GOOGLE_AI_API_KEY not found or is of the incorrect length '
|
|
1406
|
-
f'{len(GOOGLE_AI_API_KEY)}.\n'
|
|
1462
|
+
f'{len(GOOGLE_AI_API_KEY)}.\n',
|
|
1463
|
+
'',
|
|
1407
1464
|
)
|
|
1408
1465
|
|
|
1409
|
-
if 'model' not in directives:
|
|
1410
|
-
directives['model'] = 'gemini-1.5-flash-latest' # also for footer
|
|
1411
|
-
|
|
1412
1466
|
if '{unified_diff' in prompt: # matches unified_diff or unified_diff_new
|
|
1413
1467
|
default_context_lines = 9999 if '{unified_diff}' in prompt else 0 # none if only unified_diff_new
|
|
1414
1468
|
context_lines = directives.get('prompt_ud_context_lines', default_context_lines)
|
|
@@ -1425,7 +1479,7 @@ class AIGoogleDiffer(DifferBase):
|
|
|
1425
1479
|
)
|
|
1426
1480
|
if not unified_diff:
|
|
1427
1481
|
# no changes
|
|
1428
|
-
return ''
|
|
1482
|
+
return '', ''
|
|
1429
1483
|
else:
|
|
1430
1484
|
unified_diff = ''
|
|
1431
1485
|
|
|
@@ -1440,7 +1494,7 @@ class AIGoogleDiffer(DifferBase):
|
|
|
1440
1494
|
|
|
1441
1495
|
# check if data is different (same data is sent during testing)
|
|
1442
1496
|
if '{old_text}' in prompt and '{new_text}' in prompt and self.state.old_data == self.state.new_data:
|
|
1443
|
-
return ''
|
|
1497
|
+
return '', ''
|
|
1444
1498
|
|
|
1445
1499
|
model_prompt = prompt.format(
|
|
1446
1500
|
unified_diff=unified_diff,
|
|
@@ -1449,14 +1503,14 @@ class AIGoogleDiffer(DifferBase):
|
|
|
1449
1503
|
new_text=self.state.new_data,
|
|
1450
1504
|
)
|
|
1451
1505
|
|
|
1452
|
-
summary = self._send_to_model(
|
|
1506
|
+
summary, model_version = self._send_to_model(
|
|
1453
1507
|
self.job,
|
|
1454
1508
|
system_instructions,
|
|
1455
1509
|
model_prompt,
|
|
1456
1510
|
directives=directives,
|
|
1457
1511
|
)
|
|
1458
1512
|
|
|
1459
|
-
return summary
|
|
1513
|
+
return summary, model_version
|
|
1460
1514
|
|
|
1461
1515
|
if directives.get('additions_only') or self.job.additions_only:
|
|
1462
1516
|
default_system_instructions = (
|
|
@@ -1472,7 +1526,7 @@ class AIGoogleDiffer(DifferBase):
|
|
|
1472
1526
|
'You are a skilled journalist tasked with analyzing two versions of a text and summarizing the key '
|
|
1473
1527
|
'differences in meaning between them. The audience for your summary is already familiar with the '
|
|
1474
1528
|
"text's content, so you can focus on the most significant changes.\n\n"
|
|
1475
|
-
|
|
1529
|
+
'**Instructions:**\n\n'
|
|
1476
1530
|
'1. Carefully examine the old version of the text, provided within the `<old_version>` and '
|
|
1477
1531
|
'`</old_version>` tags.\n'
|
|
1478
1532
|
'2. Carefully examine the new version of the text, provided within the `<new_version>` and '
|
|
@@ -1483,25 +1537,38 @@ class AIGoogleDiffer(DifferBase):
|
|
|
1483
1537
|
'5. Summarize the identified differences, except those ignored, in a clear and concise manner, '
|
|
1484
1538
|
'explaining how the meaning has shifted or evolved in the new version compared to the old version only '
|
|
1485
1539
|
'when necessary. Be specific and provide examples to illustrate your points when needed.\n'
|
|
1486
|
-
'6. If
|
|
1540
|
+
'6. If there are only additions to the text, then summarize the additions.\n'
|
|
1487
1541
|
'7. Use Markdown formatting to structure your summary effectively. Use headings, bullet points, '
|
|
1488
1542
|
'and other Markdown elements as needed to enhance readability.\n'
|
|
1489
1543
|
'8. Restrict your analysis and summary to the information provided within the `<old_version>` and '
|
|
1490
|
-
'`<new_version
|
|
1544
|
+
'`<new_version>` tags. Do not introduce external information or assumptions.\n'
|
|
1491
1545
|
)
|
|
1492
1546
|
default_prompt = '<old_version>\n{old_text}\n</old_version>\n\n<new_version>\n{new_text}\n</new_version>'
|
|
1493
1547
|
system_instructions = directives.get('system_instructions', default_system_instructions)
|
|
1494
1548
|
prompt = directives.get('prompt', default_prompt).replace('\\n', '\n')
|
|
1495
|
-
summary = get_ai_summary(prompt, system_instructions)
|
|
1549
|
+
summary, model_version = get_ai_summary(prompt, system_instructions)
|
|
1496
1550
|
if not summary:
|
|
1497
1551
|
self.state.verb = 'changed,no_report'
|
|
1498
1552
|
return {'text': '', 'markdown': '', 'html': ''}
|
|
1499
1553
|
newline = '\n' # For Python < 3.12 f-string {} compatibility
|
|
1500
1554
|
back_n = '\\n' # For Python < 3.12 f-string {} compatibility
|
|
1501
|
-
|
|
1502
|
-
|
|
1555
|
+
directives.pop('model', None)
|
|
1556
|
+
if directives:
|
|
1557
|
+
directives_text = (
|
|
1558
|
+
' (differ directive(s): '
|
|
1559
|
+
+ (
|
|
1560
|
+
', '.join(f'{key}={str(value).replace(newline, back_n)}' for key, value in directives.items())
|
|
1561
|
+
or 'None'
|
|
1562
|
+
)
|
|
1563
|
+
+ ')'
|
|
1564
|
+
)
|
|
1565
|
+
else:
|
|
1566
|
+
directives_text = ''
|
|
1567
|
+
footer = (
|
|
1568
|
+
f"Summary by Google Generative AI's model {model_version}{directives_text}"
|
|
1569
|
+
if model_version and directives_text
|
|
1570
|
+
else ''
|
|
1503
1571
|
)
|
|
1504
|
-
footer = f'Summary generated by Google Generative AI (differ directive(s): {directives_text})'
|
|
1505
1572
|
temp_unfiltered_diff: dict[Literal['text', 'markdown', 'html'], str] = {}
|
|
1506
1573
|
for rep_kind in ['text', 'html']: # markdown is same as text
|
|
1507
1574
|
unified_report = DifferBase.process(
|
|
@@ -1513,17 +1580,16 @@ class AIGoogleDiffer(DifferBase):
|
|
|
1513
1580
|
temp_unfiltered_diff,
|
|
1514
1581
|
)
|
|
1515
1582
|
return {
|
|
1516
|
-
'text': f"{summary}\n\n{unified_report['text']}\n------------\n{footer}
|
|
1517
|
-
'markdown': f"{summary}\n\n{unified_report['markdown']}\n* * *\n{footer}
|
|
1583
|
+
'text': f"{summary}\n\n{unified_report['text']}" + (f'\n------------\n{footer}' if footer else ''),
|
|
1584
|
+
'markdown': f"{summary}\n\n{unified_report['markdown']}" + (f'\n* * *\n{footer}' if footer else ''),
|
|
1518
1585
|
'html': '\n'.join(
|
|
1519
1586
|
[
|
|
1520
1587
|
mark_to_html(summary, extras={'tables'}).replace('<h2>', '<h3>').replace('</h2>', '</h3>'),
|
|
1521
1588
|
'<br>',
|
|
1522
1589
|
'<br>',
|
|
1523
1590
|
unified_report['html'],
|
|
1524
|
-
'-----<br>',
|
|
1525
|
-
f'<i><small>{footer}</small></i>',
|
|
1526
1591
|
]
|
|
1592
|
+
+ (['-----<br>', f'<i><small>{footer}</small></i>'] if footer else [])
|
|
1527
1593
|
),
|
|
1528
1594
|
}
|
|
1529
1595
|
|
|
@@ -1575,10 +1641,13 @@ class WdiffDiffer(DifferBase):
|
|
|
1575
1641
|
add_html = '<span style="background-color:#d1ffd1;color:#082b08;">'
|
|
1576
1642
|
rem_html = '<span style="background-color:#fff0f0;color:#9c1c1c;text-decoration:line-through;">'
|
|
1577
1643
|
|
|
1578
|
-
head_text = (
|
|
1579
|
-
|
|
1580
|
-
|
|
1581
|
-
|
|
1644
|
+
head_text = '\n'.join(
|
|
1645
|
+
[
|
|
1646
|
+
# f'Differ: wdiff',
|
|
1647
|
+
f'\033[91m--- @ {self.make_timestamp(self.state.old_timestamp, tz)}\033[0m',
|
|
1648
|
+
f'\033[92m+++ @ {self.make_timestamp(self.state.new_timestamp, tz)}\033[0m',
|
|
1649
|
+
'',
|
|
1650
|
+
]
|
|
1582
1651
|
)
|
|
1583
1652
|
head_html = '<br>\n'.join(
|
|
1584
1653
|
[
|
|
@@ -194,7 +194,7 @@ class FilterBase(metaclass=TrackSubClasses):
|
|
|
194
194
|
if unknown_keys and '<any>' not in allowed_keys:
|
|
195
195
|
raise ValueError(
|
|
196
196
|
f'Job {job_index_number}: Filter {filter_kind} does not support subfilter or filter '
|
|
197
|
-
f"directive(s) {', '.join(unknown_keys)}
|
|
197
|
+
f"directive(s) {', '.join(unknown_keys)}. Only {', '.join(allowed_keys)} are supported."
|
|
198
198
|
)
|
|
199
199
|
|
|
200
200
|
yield filter_kind, subfilter
|
|
@@ -794,7 +794,10 @@ class FormatJsonFilter(FilterBase):
|
|
|
794
794
|
self.job.set_to_monospace()
|
|
795
795
|
sort_keys = subfilter.get('sort_keys', False)
|
|
796
796
|
indentation = int(subfilter.get('indentation', 4))
|
|
797
|
-
|
|
797
|
+
try:
|
|
798
|
+
parsed_json = jsonlib.loads(data)
|
|
799
|
+
except jsonlib.JSONDecodeError as e:
|
|
800
|
+
return f"Filter '{self.__kind__}' returned JSONDecodeError: {e}\n\n{data!s}", mime_type
|
|
798
801
|
if not mime_type.endswith('json'):
|
|
799
802
|
mime_type = 'application/json'
|
|
800
803
|
return jsonlib.dumps(parsed_json, ensure_ascii=False, sort_keys=sort_keys, indent=indentation), mime_type
|
|
@@ -67,7 +67,7 @@ class JobState(ContextManager):
|
|
|
67
67
|
exception: Exception | None = None
|
|
68
68
|
generated_diff: dict[Literal['text', 'markdown', 'html'], str]
|
|
69
69
|
history_dic_snapshots: dict[str | bytes, Snapshot]
|
|
70
|
-
new_data: str | bytes
|
|
70
|
+
new_data: str | bytes = ''
|
|
71
71
|
new_error_data: ErrorData = {}
|
|
72
72
|
new_etag: str
|
|
73
73
|
new_mime_type: str = ''
|
|
@@ -234,18 +234,6 @@ class JobState(ContextManager):
|
|
|
234
234
|
f'{dict(data=data, etag=self.new_etag, mime_type=mime_type)}'
|
|
235
235
|
)
|
|
236
236
|
|
|
237
|
-
# Apply automatic filters first
|
|
238
|
-
filtered_data, mime_type = FilterBase.auto_process(self, data, mime_type)
|
|
239
|
-
|
|
240
|
-
# Apply any specified filters
|
|
241
|
-
for filter_kind, subfilter in FilterBase.normalize_filter_list(self.job.filters, self.job.index_number):
|
|
242
|
-
filtered_data, mime_type = FilterBase.process(
|
|
243
|
-
filter_kind, subfilter, self, filtered_data, mime_type
|
|
244
|
-
)
|
|
245
|
-
|
|
246
|
-
self.new_data = filtered_data
|
|
247
|
-
self.new_mime_type = mime_type
|
|
248
|
-
|
|
249
237
|
except Exception as e:
|
|
250
238
|
# Job has a chance to format and ignore its error
|
|
251
239
|
if self.debugger_attached():
|
|
@@ -258,31 +246,45 @@ class JobState(ContextManager):
|
|
|
258
246
|
if not (self.error_ignored or isinstance(e, NotModifiedError)):
|
|
259
247
|
self.tries += 1
|
|
260
248
|
self.new_error_data = {
|
|
261
|
-
'type':
|
|
262
|
-
'message': str(
|
|
249
|
+
'type': e.__class__.__name__,
|
|
250
|
+
'message': str(e),
|
|
263
251
|
}
|
|
264
252
|
logger.info(
|
|
265
253
|
f'Job {self.job.index_number}: Job ended with error; incrementing cumulative error runs to '
|
|
266
254
|
f'{self.tries}'
|
|
267
255
|
)
|
|
256
|
+
|
|
257
|
+
else:
|
|
258
|
+
# Apply automatic filters first
|
|
259
|
+
filtered_data, mime_type = FilterBase.auto_process(self, data, mime_type)
|
|
260
|
+
|
|
261
|
+
# Apply any specified filters
|
|
262
|
+
for filter_kind, subfilter in FilterBase.normalize_filter_list(self.job.filters, self.job.index_number):
|
|
263
|
+
filtered_data, mime_type = FilterBase.process(
|
|
264
|
+
filter_kind, subfilter, self, filtered_data, mime_type
|
|
265
|
+
)
|
|
266
|
+
|
|
267
|
+
self.new_data = filtered_data
|
|
268
|
+
self.new_mime_type = mime_type
|
|
269
|
+
|
|
268
270
|
except Exception as e:
|
|
269
|
-
#
|
|
271
|
+
# Processing error or job failed its chance to handle error
|
|
270
272
|
if self.debugger_attached():
|
|
271
273
|
logger.warning('Running in a debugger: raising the exception instead of processing it')
|
|
272
274
|
raise
|
|
275
|
+
self.new_timestamp = time.time()
|
|
273
276
|
self.exception = e
|
|
274
|
-
self.traceback =
|
|
277
|
+
self.traceback = ''.join(traceback.format_exception_only(e, show_group=True)).rstrip()
|
|
275
278
|
self.error_ignored = False
|
|
276
|
-
|
|
277
|
-
|
|
278
|
-
|
|
279
|
-
|
|
280
|
-
|
|
281
|
-
|
|
282
|
-
|
|
283
|
-
|
|
284
|
-
|
|
285
|
-
)
|
|
279
|
+
self.tries += 1
|
|
280
|
+
self.new_error_data = {
|
|
281
|
+
'type': '.'.join(filter(None, [getattr(e, '__module__', None), e.__class__.__name__])),
|
|
282
|
+
'message': str(e),
|
|
283
|
+
}
|
|
284
|
+
logger.info(
|
|
285
|
+
f'Job {self.job.index_number}: Job ended with error (internal handling failed); incrementing '
|
|
286
|
+
f'cumulative error runs to {self.tries}'
|
|
287
|
+
)
|
|
286
288
|
|
|
287
289
|
logger.debug(f'Job {self.job.index_number}: Processed as {self.added_data()}')
|
|
288
290
|
logger.info(f'{self.job.get_indexed_location()} ended processing')
|
|
@@ -323,7 +325,7 @@ class JobState(ContextManager):
|
|
|
323
325
|
_generated_diff, _mime_type = FilterBase.process( # type: ignore[assignment]
|
|
324
326
|
filter_kind, subfilter, self, _generated_diff, _mime_type
|
|
325
327
|
)
|
|
326
|
-
self.generated_diff[report_kind] = _generated_diff
|
|
328
|
+
self.generated_diff[report_kind] = str(_generated_diff)
|
|
327
329
|
|
|
328
330
|
return self.generated_diff[report_kind]
|
|
329
331
|
|
|
@@ -786,7 +786,6 @@ class UrlJob(UrlJobBase):
|
|
|
786
786
|
http_error_msg = f'{response.status_code} Client Error: {reason} for url: {response.url}'
|
|
787
787
|
else:
|
|
788
788
|
http_error_msg = f'{response.status_code} Server Error: {reason} for url: {response.url}'
|
|
789
|
-
logger.error(f'httpx received {http_error_msg}')
|
|
790
789
|
|
|
791
790
|
if response.status_code != 404:
|
|
792
791
|
try:
|
|
@@ -858,8 +857,7 @@ class UrlJob(UrlJobBase):
|
|
|
858
857
|
urllib3.util.ssl_.DEFAULT_CIPHERS += 'HIGH:!DH:!aNULL' # type: ignore[attr-defined]
|
|
859
858
|
except AttributeError:
|
|
860
859
|
logger.error(
|
|
861
|
-
'Unable to ignore_dh_key_too_small due to bug in '
|
|
862
|
-
'requests.packages.urrlib3.util.ssl.DEFAULT_CIPHERS'
|
|
860
|
+
'Unable to ignore_dh_key_too_small due to bug in requests.packages.urrlib3.util.ssl.DEFAULT_CIPHERS'
|
|
863
861
|
)
|
|
864
862
|
logger.error('See https://github.com/psf/requests/issues/6443')
|
|
865
863
|
|
|
@@ -1105,7 +1103,6 @@ class UrlJob(UrlJobBase):
|
|
|
1105
1103
|
):
|
|
1106
1104
|
# Instead of a full traceback, just show the error
|
|
1107
1105
|
exception_str = str(exception).strip()
|
|
1108
|
-
print(f'{exception_str=} {exception.args=} {type(exception)=}')
|
|
1109
1106
|
if self.proxy and (
|
|
1110
1107
|
(httpx and isinstance(exception, httpx.TransportError))
|
|
1111
1108
|
or any(
|
|
@@ -1373,7 +1370,7 @@ class BrowserJob(UrlJobBase):
|
|
|
1373
1370
|
ignore_default_args=ignore_default_args,
|
|
1374
1371
|
timeout=timeout,
|
|
1375
1372
|
headless=headless,
|
|
1376
|
-
proxy=proxy,
|
|
1373
|
+
proxy=proxy, # type: ignore[arg-type]
|
|
1377
1374
|
)
|
|
1378
1375
|
browser_version = browser.version
|
|
1379
1376
|
user_agent = headers.pop(
|
|
@@ -1402,7 +1399,7 @@ class BrowserJob(UrlJobBase):
|
|
|
1402
1399
|
args=args,
|
|
1403
1400
|
ignore_default_args=ignore_default_args,
|
|
1404
1401
|
headless=headless,
|
|
1405
|
-
proxy=proxy,
|
|
1402
|
+
proxy=proxy, # type: ignore[arg-type]
|
|
1406
1403
|
no_viewport=no_viewport,
|
|
1407
1404
|
ignore_https_errors=self.ignore_https_errors,
|
|
1408
1405
|
extra_http_headers=dict(headers),
|
|
@@ -1811,7 +1808,7 @@ class BrowserJob(UrlJobBase):
|
|
|
1811
1808
|
:param tb: The traceback.format_exc() string.
|
|
1812
1809
|
:returns: A string to display and/or use in reports.
|
|
1813
1810
|
"""
|
|
1814
|
-
exception_str =
|
|
1811
|
+
exception_str = str(exception).strip()
|
|
1815
1812
|
print(f'{exception_str=}, {tb=}')
|
|
1816
1813
|
if self.proxy and 'net::ERR' in exception_str:
|
|
1817
1814
|
exception_str += f'\n\n(Job has proxy {self.proxy})'
|
|
@@ -117,7 +117,7 @@ if sys.platform == 'win32':
|
|
|
117
117
|
try:
|
|
118
118
|
from colorama import AnsiToWin32
|
|
119
119
|
except ImportError as e: # pragma: no cover
|
|
120
|
-
AnsiToWin32 = str(e) # type: ignore[assignment]
|
|
120
|
+
AnsiToWin32 = str(e) # type: ignore[assignment,misc]
|
|
121
121
|
|
|
122
122
|
logger = logging.getLogger(__name__)
|
|
123
123
|
|
|
@@ -405,7 +405,7 @@ class HtmlReporter(ReporterBase):
|
|
|
405
405
|
[mark_to_html(line, job_state.job.markdown_padded_tables) for line in text.splitlines()]
|
|
406
406
|
)
|
|
407
407
|
|
|
408
|
-
if job_state.verb == 'error':
|
|
408
|
+
if job_state.verb == 'error' or job_state.verb == 'repeated_error':
|
|
409
409
|
htm = f'<pre style="white-space:pre-wrap;color:red;">{html.escape(job_state.traceback)}</pre>'
|
|
410
410
|
if job_state.job.suppress_repeated_errors:
|
|
411
411
|
htm += (
|
|
@@ -518,7 +518,7 @@ class TextReporter(ReporterBase):
|
|
|
518
518
|
:param differ: The type of differ to use.
|
|
519
519
|
:returns: HTML for a single job.
|
|
520
520
|
"""
|
|
521
|
-
if job_state.verb == 'error':
|
|
521
|
+
if job_state.verb == 'error' or job_state.verb == 'repeated_error':
|
|
522
522
|
if isinstance(self, StdoutReporter):
|
|
523
523
|
text = self._red(job_state.traceback)
|
|
524
524
|
else:
|
|
@@ -780,7 +780,7 @@ class MarkdownReporter(ReporterBase):
|
|
|
780
780
|
:param differ: The type of differ to use.
|
|
781
781
|
:returns: HTML for a single job.
|
|
782
782
|
"""
|
|
783
|
-
if job_state.verb == 'error':
|
|
783
|
+
if job_state.verb == 'error' or job_state.verb == 'repeated_error':
|
|
784
784
|
mark = job_state.traceback
|
|
785
785
|
if job_state.job.suppress_repeated_errors:
|
|
786
786
|
mark += '_Reminder: No further alerts until the error is resolved or changes._'
|
|
@@ -798,8 +798,8 @@ class YamlConfigStorage(BaseYamlFileStorage):
|
|
|
798
798
|
)
|
|
799
799
|
else:
|
|
800
800
|
config['job_defaults']['command'] = config['job_defaults'].pop(
|
|
801
|
-
'shell'
|
|
802
|
-
)
|
|
801
|
+
'shell' # type: ignore[typeddict-item]
|
|
802
|
+
)
|
|
803
803
|
|
|
804
804
|
for key in {'all', 'url', 'browser', 'command'}:
|
|
805
805
|
if key not in config['job_defaults']:
|
|
@@ -1,6 +1,6 @@
|
|
|
1
|
-
Metadata-Version: 2.
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
2
|
Name: webchanges
|
|
3
|
-
Version: 3.
|
|
3
|
+
Version: 3.29.0
|
|
4
4
|
Summary: Web Changes Delivered. AI-Summarized. Totally Anonymous.
|
|
5
5
|
Author-email: Mike Borsetti <mike+webchanges@borsetti.com>
|
|
6
6
|
Maintainer-email: Mike Borsetti <mike+webchanges@borsetti.com>
|
|
@@ -163,6 +163,7 @@ Provides-Extra: safe-password
|
|
|
163
163
|
Requires-Dist: keyring; extra == "safe-password"
|
|
164
164
|
Provides-Extra: all
|
|
165
165
|
Requires-Dist: webchanges[beautify,bs4,deepdiff_xml,html5lib,ical2text,imagediff,jq,matrix,ocr,pdf2text,pushbullet,pushover,pypdf_crypto,redis,requests,safe_password,use_browser,xmpp]; extra == "all"
|
|
166
|
+
Dynamic: license-file
|
|
166
167
|
|
|
167
168
|
.. role:: underline
|
|
168
169
|
:class: underline
|
|
@@ -195,7 +196,9 @@ For Generative AI summaries (BETA), you need a free `API Key from Google Cloud A
|
|
|
195
196
|
|
|
196
197
|
Installation
|
|
197
198
|
============
|
|
198
|
-
|
|
199
|
+
|pypi_version| |format| |status| |security|
|
|
200
|
+
|
|
201
|
+
Install **webchanges** with:
|
|
199
202
|
|
|
200
203
|
.. code-block:: bash
|
|
201
204
|
|
|
@@ -203,7 +206,7 @@ Install **webchanges** |pypi_version| |format| |status| |security| with:
|
|
|
203
206
|
|
|
204
207
|
Running in Docker
|
|
205
208
|
-----------------
|
|
206
|
-
**webchanges** can easily run in a container
|
|
209
|
+
**webchanges** can easily run in a container and you will find a `Docker <https://www.docker.com/>`__ implementation
|
|
207
210
|
`here <https://github.com/yubiuser/webchanges-docker>`__.
|
|
208
211
|
|
|
209
212
|
|
|
@@ -247,20 +250,19 @@ Schedule
|
|
|
247
250
|
--------
|
|
248
251
|
**webchanges** leverages the power of a system scheduler:
|
|
249
252
|
|
|
250
|
-
- On Linux you can use cron,
|
|
251
|
-
|
|
252
|
-
used cron before);
|
|
253
|
+
- On Linux you can use cron, with the help of a tool like `crontab.guru <https://crontab.guru>`__ (help `here
|
|
254
|
+
<https://www.computerhope.com/unix/ucrontab.htm>`__);
|
|
253
255
|
- On Windows you can use `Windows Task Scheduler <https://en.wikipedia.org/wiki/Windows_Task_Scheduler>`__;
|
|
254
256
|
- On macOS you can use `launchd <https://developer.apple
|
|
255
|
-
.com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/ScheduledJobs.html>`__ (
|
|
256
|
-
<https://launchd.info/>`__
|
|
257
|
+
.com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/ScheduledJobs.html>`__ (help `here
|
|
258
|
+
<https://launchd.info/>`__).
|
|
257
259
|
|
|
258
260
|
|
|
259
261
|
Code
|
|
260
262
|
====
|
|
261
|
-
|coveralls| |issues|
|
|
263
|
+
|coveralls| |issues| |code_style|
|
|
262
264
|
|
|
263
|
-
The code
|
|
265
|
+
The code, issues tracker, and discussions are hosted on `GitHub <https://github.com/mborsetti/webchanges>`__.
|
|
264
266
|
|
|
265
267
|
|
|
266
268
|
Contributing
|
|
@@ -344,7 +346,7 @@ Example enhancements to HTML reporting:
|
|
|
344
346
|
.. |format| image:: https://img.shields.io/pypi/format/webchanges.svg
|
|
345
347
|
:target: https://pypi.org/project/webchanges/
|
|
346
348
|
:alt: Kit format
|
|
347
|
-
.. |downloads| image:: https://
|
|
349
|
+
.. |downloads| image:: https://img.shields.io/pypi/dm/webchanges.svg
|
|
348
350
|
:target: https://www.pepy.tech/project/webchanges
|
|
349
351
|
:alt: PyPI downloads
|
|
350
352
|
.. |license| image:: https://img.shields.io/pypi/l/webchanges.svg
|
|
@@ -356,15 +358,24 @@ Example enhancements to HTML reporting:
|
|
|
356
358
|
.. |readthedocs| image:: https://img.shields.io/readthedocs/webchanges/stable.svg?label=
|
|
357
359
|
:target: https://webchanges.readthedocs.io/
|
|
358
360
|
:alt: Documentation status
|
|
359
|
-
.. |
|
|
361
|
+
.. |old_CI| image:: https://github.com/mborsetti/webchanges/actions/workflows/ci-cd.yaml/badge.svg?event=push
|
|
362
|
+
:target: https://github.com/mborsetti/webchanges/actions
|
|
363
|
+
:alt: CI testing status
|
|
364
|
+
.. |CI| image:: https://img.shields.io/github/check-runs/mborsetti/webchanges/main
|
|
360
365
|
:target: https://github.com/mborsetti/webchanges/actions
|
|
361
366
|
:alt: CI testing status
|
|
362
|
-
.. |
|
|
367
|
+
.. |old_coveralls| image:: https://coveralls.io/repos/github/mborsetti/webchanges/badge.svg?branch=main
|
|
368
|
+
:target: https://coveralls.io/github/mborsetti/webchanges?branch=main
|
|
369
|
+
:alt: Code coverage by Coveralls
|
|
370
|
+
.. |coveralls| image:: https://img.shields.io/coverallsCoverage/github/mborsetti/webchanges.svg
|
|
363
371
|
:target: https://coveralls.io/github/mborsetti/webchanges?branch=main
|
|
364
372
|
:alt: Code coverage by Coveralls
|
|
373
|
+
.. |code_style| image:: https://img.shields.io/badge/code%20style-black-000000.svg
|
|
374
|
+
:target: https://github.com/psf/black
|
|
375
|
+
:alt: Code style black
|
|
365
376
|
.. |status| image:: https://img.shields.io/pypi/status/webchanges.svg
|
|
366
377
|
:target: https://pypi.org/project/webchanges/
|
|
367
378
|
:alt: Package stability
|
|
368
|
-
.. |security| image:: https://img.shields.io/badge/security-bandit-
|
|
379
|
+
.. |security| image:: https://img.shields.io/badge/security-bandit-green.svg
|
|
369
380
|
:target: https://github.com/PyCQA/bandit
|
|
370
381
|
:alt: Security Status
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|