webchanges 3.28.1__tar.gz → 3.29.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (33) hide show
  1. {webchanges-3.28.1/webchanges.egg-info → webchanges-3.29.0}/PKG-INFO +26 -15
  2. {webchanges-3.28.1 → webchanges-3.29.0}/README.rst +23 -13
  3. {webchanges-3.28.1 → webchanges-3.29.0}/webchanges/__init__.py +1 -1
  4. webchanges-3.29.0/webchanges/__main__.py +10 -0
  5. {webchanges-3.28.1 → webchanges-3.29.0}/webchanges/cli.py +11 -7
  6. {webchanges-3.28.1 → webchanges-3.29.0}/webchanges/command.py +2 -2
  7. {webchanges-3.28.1 → webchanges-3.29.0}/webchanges/differs.py +152 -83
  8. {webchanges-3.28.1 → webchanges-3.29.0}/webchanges/filters.py +5 -2
  9. {webchanges-3.28.1 → webchanges-3.29.0}/webchanges/handler.py +30 -28
  10. {webchanges-3.28.1 → webchanges-3.29.0}/webchanges/jobs.py +4 -7
  11. {webchanges-3.28.1 → webchanges-3.29.0}/webchanges/reporters.py +4 -4
  12. {webchanges-3.28.1 → webchanges-3.29.0}/webchanges/storage.py +2 -2
  13. {webchanges-3.28.1 → webchanges-3.29.0/webchanges.egg-info}/PKG-INFO +26 -15
  14. {webchanges-3.28.1 → webchanges-3.29.0}/webchanges.egg-info/SOURCES.txt +1 -0
  15. {webchanges-3.28.1 → webchanges-3.29.0}/LICENSE +0 -0
  16. {webchanges-3.28.1 → webchanges-3.29.0}/MANIFEST.in +0 -0
  17. {webchanges-3.28.1 → webchanges-3.29.0}/pyproject.toml +0 -0
  18. {webchanges-3.28.1 → webchanges-3.29.0}/requirements.txt +0 -0
  19. {webchanges-3.28.1 → webchanges-3.29.0}/setup.cfg +0 -0
  20. {webchanges-3.28.1 → webchanges-3.29.0}/webchanges/_vendored/__init__.py +0 -0
  21. {webchanges-3.28.1 → webchanges-3.29.0}/webchanges/_vendored/headers.py +0 -0
  22. {webchanges-3.28.1 → webchanges-3.29.0}/webchanges/_vendored/packaging_version.py +0 -0
  23. {webchanges-3.28.1 → webchanges-3.29.0}/webchanges/config.py +0 -0
  24. {webchanges-3.28.1 → webchanges-3.29.0}/webchanges/mailer.py +0 -0
  25. {webchanges-3.28.1 → webchanges-3.29.0}/webchanges/main.py +0 -0
  26. {webchanges-3.28.1 → webchanges-3.29.0}/webchanges/py.typed +0 -0
  27. {webchanges-3.28.1 → webchanges-3.29.0}/webchanges/storage_minidb.py +0 -0
  28. {webchanges-3.28.1 → webchanges-3.29.0}/webchanges/util.py +0 -0
  29. {webchanges-3.28.1 → webchanges-3.29.0}/webchanges/worker.py +0 -0
  30. {webchanges-3.28.1 → webchanges-3.29.0}/webchanges.egg-info/dependency_links.txt +0 -0
  31. {webchanges-3.28.1 → webchanges-3.29.0}/webchanges.egg-info/entry_points.txt +0 -0
  32. {webchanges-3.28.1 → webchanges-3.29.0}/webchanges.egg-info/requires.txt +0 -0
  33. {webchanges-3.28.1 → webchanges-3.29.0}/webchanges.egg-info/top_level.txt +0 -0
@@ -1,6 +1,6 @@
1
- Metadata-Version: 2.2
1
+ Metadata-Version: 2.4
2
2
  Name: webchanges
3
- Version: 3.28.1
3
+ Version: 3.29.0
4
4
  Summary: Web Changes Delivered. AI-Summarized. Totally Anonymous.
5
5
  Author-email: Mike Borsetti <mike+webchanges@borsetti.com>
6
6
  Maintainer-email: Mike Borsetti <mike+webchanges@borsetti.com>
@@ -163,6 +163,7 @@ Provides-Extra: safe-password
163
163
  Requires-Dist: keyring; extra == "safe-password"
164
164
  Provides-Extra: all
165
165
  Requires-Dist: webchanges[beautify,bs4,deepdiff_xml,html5lib,ical2text,imagediff,jq,matrix,ocr,pdf2text,pushbullet,pushover,pypdf_crypto,redis,requests,safe_password,use_browser,xmpp]; extra == "all"
166
+ Dynamic: license-file
166
167
 
167
168
  .. role:: underline
168
169
  :class: underline
@@ -195,7 +196,9 @@ For Generative AI summaries (BETA), you need a free `API Key from Google Cloud A
195
196
 
196
197
  Installation
197
198
  ============
198
- Install **webchanges** |pypi_version| |format| |status| |security| with:
199
+ |pypi_version| |format| |status| |security|
200
+
201
+ Install **webchanges** with:
199
202
 
200
203
  .. code-block:: bash
201
204
 
@@ -203,7 +206,7 @@ Install **webchanges** |pypi_version| |format| |status| |security| with:
203
206
 
204
207
  Running in Docker
205
208
  -----------------
206
- **webchanges** can easily run in a container; you can find a `Docker <https://www.docker.com/>`__ implementation
209
+ **webchanges** can easily run in a container and you will find a `Docker <https://www.docker.com/>`__ implementation
207
210
  `here <https://github.com/yubiuser/webchanges-docker>`__.
208
211
 
209
212
 
@@ -247,20 +250,19 @@ Schedule
247
250
  --------
248
251
  **webchanges** leverages the power of a system scheduler:
249
252
 
250
- - On Linux you can use cron, and a tool like `crontab.guru <https://crontab.guru>`__ can build a
251
- schedule expression for you (note: see `here <https://www.computerhope.com/unix/ucrontab.htm>`__ if you have never
252
- used cron before);
253
+ - On Linux you can use cron, with the help of a tool like `crontab.guru <https://crontab.guru>`__ (help `here
254
+ <https://www.computerhope.com/unix/ucrontab.htm>`__);
253
255
  - On Windows you can use `Windows Task Scheduler <https://en.wikipedia.org/wiki/Windows_Task_Scheduler>`__;
254
256
  - On macOS you can use `launchd <https://developer.apple
255
- .com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/ScheduledJobs.html>`__ (note: see `here
256
- <https://launchd.info/>`__ if you have never used launchd before).
257
+ .com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/ScheduledJobs.html>`__ (help `here
258
+ <https://launchd.info/>`__).
257
259
 
258
260
 
259
261
  Code
260
262
  ====
261
- |coveralls| |issues|
263
+ |coveralls| |issues| |code_style|
262
264
 
263
- The code and issues tracker are hosted on `GitHub <https://github.com/mborsetti/webchanges>`__.
265
+ The code, issues tracker, and discussions are hosted on `GitHub <https://github.com/mborsetti/webchanges>`__.
264
266
 
265
267
 
266
268
  Contributing
@@ -344,7 +346,7 @@ Example enhancements to HTML reporting:
344
346
  .. |format| image:: https://img.shields.io/pypi/format/webchanges.svg
345
347
  :target: https://pypi.org/project/webchanges/
346
348
  :alt: Kit format
347
- .. |downloads| image:: https://static.pepy.tech/badge/webchanges
349
+ .. |downloads| image:: https://img.shields.io/pypi/dm/webchanges.svg
348
350
  :target: https://www.pepy.tech/project/webchanges
349
351
  :alt: PyPI downloads
350
352
  .. |license| image:: https://img.shields.io/pypi/l/webchanges.svg
@@ -356,15 +358,24 @@ Example enhancements to HTML reporting:
356
358
  .. |readthedocs| image:: https://img.shields.io/readthedocs/webchanges/stable.svg?label=
357
359
  :target: https://webchanges.readthedocs.io/
358
360
  :alt: Documentation status
359
- .. |CI| image:: https://github.com/mborsetti/webchanges/actions/workflows/ci-cd.yaml/badge.svg?event=push
361
+ .. |old_CI| image:: https://github.com/mborsetti/webchanges/actions/workflows/ci-cd.yaml/badge.svg?event=push
362
+ :target: https://github.com/mborsetti/webchanges/actions
363
+ :alt: CI testing status
364
+ .. |CI| image:: https://img.shields.io/github/check-runs/mborsetti/webchanges/main
360
365
  :target: https://github.com/mborsetti/webchanges/actions
361
366
  :alt: CI testing status
362
- .. |coveralls| image:: https://coveralls.io/repos/github/mborsetti/webchanges/badge.svg?branch=main
367
+ .. |old_coveralls| image:: https://coveralls.io/repos/github/mborsetti/webchanges/badge.svg?branch=main
368
+ :target: https://coveralls.io/github/mborsetti/webchanges?branch=main
369
+ :alt: Code coverage by Coveralls
370
+ .. |coveralls| image:: https://img.shields.io/coverallsCoverage/github/mborsetti/webchanges.svg
363
371
  :target: https://coveralls.io/github/mborsetti/webchanges?branch=main
364
372
  :alt: Code coverage by Coveralls
373
+ .. |code_style| image:: https://img.shields.io/badge/code%20style-black-000000.svg
374
+ :target: https://github.com/psf/black
375
+ :alt: Code style black
365
376
  .. |status| image:: https://img.shields.io/pypi/status/webchanges.svg
366
377
  :target: https://pypi.org/project/webchanges/
367
378
  :alt: Package stability
368
- .. |security| image:: https://img.shields.io/badge/security-bandit-yellow.svg
379
+ .. |security| image:: https://img.shields.io/badge/security-bandit-green.svg
369
380
  :target: https://github.com/PyCQA/bandit
370
381
  :alt: Security Status
@@ -29,7 +29,9 @@ For Generative AI summaries (BETA), you need a free `API Key from Google Cloud A
29
29
 
30
30
  Installation
31
31
  ============
32
- Install **webchanges** |pypi_version| |format| |status| |security| with:
32
+ |pypi_version| |format| |status| |security|
33
+
34
+ Install **webchanges** with:
33
35
 
34
36
  .. code-block:: bash
35
37
 
@@ -37,7 +39,7 @@ Install **webchanges** |pypi_version| |format| |status| |security| with:
37
39
 
38
40
  Running in Docker
39
41
  -----------------
40
- **webchanges** can easily run in a container; you can find a `Docker <https://www.docker.com/>`__ implementation
42
+ **webchanges** can easily run in a container and you will find a `Docker <https://www.docker.com/>`__ implementation
41
43
  `here <https://github.com/yubiuser/webchanges-docker>`__.
42
44
 
43
45
 
@@ -81,20 +83,19 @@ Schedule
81
83
  --------
82
84
  **webchanges** leverages the power of a system scheduler:
83
85
 
84
- - On Linux you can use cron, and a tool like `crontab.guru <https://crontab.guru>`__ can build a
85
- schedule expression for you (note: see `here <https://www.computerhope.com/unix/ucrontab.htm>`__ if you have never
86
- used cron before);
86
+ - On Linux you can use cron, with the help of a tool like `crontab.guru <https://crontab.guru>`__ (help `here
87
+ <https://www.computerhope.com/unix/ucrontab.htm>`__);
87
88
  - On Windows you can use `Windows Task Scheduler <https://en.wikipedia.org/wiki/Windows_Task_Scheduler>`__;
88
89
  - On macOS you can use `launchd <https://developer.apple
89
- .com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/ScheduledJobs.html>`__ (note: see `here
90
- <https://launchd.info/>`__ if you have never used launchd before).
90
+ .com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/ScheduledJobs.html>`__ (help `here
91
+ <https://launchd.info/>`__).
91
92
 
92
93
 
93
94
  Code
94
95
  ====
95
- |coveralls| |issues|
96
+ |coveralls| |issues| |code_style|
96
97
 
97
- The code and issues tracker are hosted on `GitHub <https://github.com/mborsetti/webchanges>`__.
98
+ The code, issues tracker, and discussions are hosted on `GitHub <https://github.com/mborsetti/webchanges>`__.
98
99
 
99
100
 
100
101
  Contributing
@@ -178,7 +179,7 @@ Example enhancements to HTML reporting:
178
179
  .. |format| image:: https://img.shields.io/pypi/format/webchanges.svg
179
180
  :target: https://pypi.org/project/webchanges/
180
181
  :alt: Kit format
181
- .. |downloads| image:: https://static.pepy.tech/badge/webchanges
182
+ .. |downloads| image:: https://img.shields.io/pypi/dm/webchanges.svg
182
183
  :target: https://www.pepy.tech/project/webchanges
183
184
  :alt: PyPI downloads
184
185
  .. |license| image:: https://img.shields.io/pypi/l/webchanges.svg
@@ -190,15 +191,24 @@ Example enhancements to HTML reporting:
190
191
  .. |readthedocs| image:: https://img.shields.io/readthedocs/webchanges/stable.svg?label=
191
192
  :target: https://webchanges.readthedocs.io/
192
193
  :alt: Documentation status
193
- .. |CI| image:: https://github.com/mborsetti/webchanges/actions/workflows/ci-cd.yaml/badge.svg?event=push
194
+ .. |old_CI| image:: https://github.com/mborsetti/webchanges/actions/workflows/ci-cd.yaml/badge.svg?event=push
195
+ :target: https://github.com/mborsetti/webchanges/actions
196
+ :alt: CI testing status
197
+ .. |CI| image:: https://img.shields.io/github/check-runs/mborsetti/webchanges/main
194
198
  :target: https://github.com/mborsetti/webchanges/actions
195
199
  :alt: CI testing status
196
- .. |coveralls| image:: https://coveralls.io/repos/github/mborsetti/webchanges/badge.svg?branch=main
200
+ .. |old_coveralls| image:: https://coveralls.io/repos/github/mborsetti/webchanges/badge.svg?branch=main
201
+ :target: https://coveralls.io/github/mborsetti/webchanges?branch=main
202
+ :alt: Code coverage by Coveralls
203
+ .. |coveralls| image:: https://img.shields.io/coverallsCoverage/github/mborsetti/webchanges.svg
197
204
  :target: https://coveralls.io/github/mborsetti/webchanges?branch=main
198
205
  :alt: Code coverage by Coveralls
206
+ .. |code_style| image:: https://img.shields.io/badge/code%20style-black-000000.svg
207
+ :target: https://github.com/psf/black
208
+ :alt: Code style black
199
209
  .. |status| image:: https://img.shields.io/pypi/status/webchanges.svg
200
210
  :target: https://pypi.org/project/webchanges/
201
211
  :alt: Package stability
202
- .. |security| image:: https://img.shields.io/badge/security-bandit-yellow.svg
212
+ .. |security| image:: https://img.shields.io/badge/security-bandit-green.svg
203
213
  :target: https://github.com/PyCQA/bandit
204
214
  :alt: Security Status
@@ -22,7 +22,7 @@ __project_name__ = __package__
22
22
  # * MINOR version when you add functionality in a backwards compatible manner, and
23
23
  # * MICRO or PATCH version when you make backwards compatible bug fixes. We no longer use '0'
24
24
  # If unsure on increments, use pkg_resources.parse_version to parse
25
- __version__ = '3.28.1'
25
+ __version__ = '3.29.0'
26
26
  __description__ = (
27
27
  'Check web (or command output) for changes since last run and notify.\n'
28
28
  '\n'
@@ -0,0 +1,10 @@
1
+ import sys
2
+ from pathlib import Path
3
+
4
+ parent_dir = Path(__file__).parent.parent
5
+ sys.path.insert(1, str(parent_dir))
6
+
7
+ if __name__ == '__main__':
8
+ from cli import main
9
+
10
+ main()
@@ -244,19 +244,18 @@ def first_run(command_config: CommandConfig) -> None:
244
244
  def load_hooks(hooks_file: Path) -> None:
245
245
  """Load hooks file."""
246
246
  if not hooks_file.is_file():
247
- warnings.warn(
248
- f'Hooks file not imported because {hooks_file} is not a file',
249
- ImportWarning,
250
- )
247
+ # do not use ImportWarning as it could be suppressed
248
+ warnings.warn(f'Hooks file not imported because {hooks_file} is not a file', RuntimeWarning)
251
249
  return
252
250
 
253
251
  hooks_file_errors = file_ownership_checks(hooks_file)
254
252
  if hooks_file_errors:
253
+ logger.debug('Here should come the warning')
254
+ # do not use ImportWarning as it could be suppressed
255
255
  warnings.warn(
256
- f'Hooks file {hooks_file} not imported because '
257
- f" {' and '.join(hooks_file_errors)}.\n"
256
+ f"Hooks file {hooks_file} not not imported because{' and '.join(hooks_file_errors)}.\n"
258
257
  f'(see {__docs_url__}en/stable/hooks.html#important-note-for-hooks-file)',
259
- ImportWarning,
258
+ RuntimeWarning,
260
259
  )
261
260
  else:
262
261
  logger.info(f'Importing hooks module from {hooks_file}')
@@ -372,6 +371,10 @@ def main() -> None: # pragma: no cover
372
371
  # Set up the logger to verbose if needed
373
372
  setup_logger(command_config.verbose, command_config.log_file)
374
373
 
374
+ # log defaults
375
+ logger.debug(f'Default config path is {config_path}')
376
+ logger.debug(f'Default data path is {data_path}')
377
+
375
378
  # For speed, run these here
376
379
  handle_unitialized_actions(command_config)
377
380
 
@@ -401,6 +404,7 @@ def main() -> None: # pragma: no cover
401
404
 
402
405
  # load config (which for syntax checking requires hooks to be loaded too)
403
406
  if command_config.hooks_files:
407
+ logger.debug(f'Hooks files to be loaded: {command_config.hooks_files}')
404
408
  for hooks_file in command_config.hooks_files:
405
409
  load_hooks(hooks_file)
406
410
  config_storage.load()
@@ -498,8 +498,8 @@ class UrlwatchCommand:
498
498
 
499
499
  def test_differ(self, arg_test_differ: list[str]) -> int:
500
500
  """
501
- Runs diffs for a job on all the saved snapshots and outputs the result to stdout or whatever reporter is
502
- selected with --test-reporter.
501
+ Runs diffs for a job on all the saved snapshots and outputs the result to stdout or the reporter selected
502
+ with --test-reporter.
503
503
 
504
504
  :param arg_test_differ: Either the job_id or a list containing [job_id, max_diffs]
505
505
  :return: 1 if error, 0 if successful.
@@ -151,12 +151,13 @@ class DifferBase(metaclass=TrackSubClasses):
151
151
  ) -> dict[str, Any]:
152
152
  """Obtain differ subdirectives that also contains defaults from the configuration.
153
153
 
154
- :param differ_kind: The differ kind.
154
+ :param differ_spec: The differ as entered by the user; use "unified" if empty.
155
155
  :param directives: The differ directives as stated in the job.
156
+ :param config: The configuration.
156
157
  :returns: directives inclusive of configuration defaults.
157
158
  """
158
159
  if config is None:
159
- logger.error('Cannot merge differ differdirectives with defaults as no config object was passed')
160
+ logger.info('No configuration object found to look for differ defaults')
160
161
  return directives
161
162
  cfg = config.get('differ_defaults')
162
163
  if isinstance(cfg, dict):
@@ -261,9 +262,13 @@ class DifferBase(metaclass=TrackSubClasses):
261
262
  _unfiltered_diff: dict[Literal['text', 'markdown', 'html'], str] | None = None,
262
263
  tz: ZoneInfo | None = None,
263
264
  ) -> dict[Literal['text', 'markdown', 'html'], str]:
264
- """Create a diff from the data. Since this function could be called by different reporters of multiple report
265
- types ('text', 'markdown', 'html'), the differ outputs a dict with data for the report_kind it generated so
266
- that it can be reused.
265
+ """Generate a formatted diff representation of data changes.
266
+
267
+ Creates a diff representation in one or more output formats (text, markdown, or HTML).
268
+ At minimum, this function must return output in the format specified by 'report_kind'.
269
+ As results are memoized for performance optimization, it can generate up to all three formats simultaneously.
270
+
271
+ :param state: The JobState.
267
272
 
268
273
  :param directives: The directives.
269
274
  :param report_kind: The report_kind for which a diff must be generated (at a minimum).
@@ -281,12 +286,14 @@ class DifferBase(metaclass=TrackSubClasses):
281
286
  timestamp: float,
282
287
  tz: ZoneInfo | None = None,
283
288
  ) -> str:
284
- """Creates a datetime string in RFC 5322 (email) format with the time zone name (if available) in the
285
- Comments and Folding White Space (CFWS) section.
289
+ """Format a timestamp as an RFC 5322 compliant datetime string.
290
+
291
+ Converts a numeric timestamp to a formatted datetime string following the RFC 5322 (email) standard. When a
292
+ timezone is provided, its full name, if known, is appended.
286
293
 
287
294
  :param timestamp: The timestamp.
288
295
  :param tz: The IANA timezone of the report.
289
- :returns: A datetime string in RFC 5322 (email) format.
296
+ :returns: A datetime string in RFC 5322 (email) format or 'NEW' if timestamp is 0.
290
297
  """
291
298
  if timestamp:
292
299
  dt = datetime.fromtimestamp(timestamp).astimezone(tz=tz)
@@ -549,6 +556,7 @@ class CommandDiffer(DifferBase):
549
556
 
550
557
  __supported_directives__ = {
551
558
  'command': 'The command to execute',
559
+ 'is_html': 'Whether the output of the command is HTML',
552
560
  }
553
561
 
554
562
  re_ptags = re.compile(r'^<p>|</p>$')
@@ -562,15 +570,30 @@ class CommandDiffer(DifferBase):
562
570
  _unfiltered_diff: dict[Literal['text', 'markdown', 'html'], str] | None = None,
563
571
  tz: ZoneInfo | None = None,
564
572
  ) -> dict[Literal['text', 'markdown', 'html'], str]:
573
+ if self.job.monospace:
574
+ head_html = '\n'.join(
575
+ [
576
+ '<span style="font-family:monospace;white-space:pre-wrap;">',
577
+ # f"Using command differ: {directives['command']}",
578
+ f'<span style="color:darkred;">--- @ {self.make_timestamp(self.state.old_timestamp, tz)}</span>',
579
+ f'<span style="color:darkgreen;">+++ @ {self.make_timestamp(self.state.new_timestamp, tz)}</span>',
580
+ ]
581
+ )
582
+ else:
583
+ head_html = '<br>\n'.join(
584
+ [
585
+ '<span style="font-family:monospace;">',
586
+ # f"Using command differ: {directives['command']}",
587
+ f'<span style="color:darkred;">--- @ {self.make_timestamp(self.state.old_timestamp, tz)}</span>',
588
+ f'<span style="color:darkgreen;">+++ @ {self.make_timestamp(self.state.new_timestamp, tz)}</span>',
589
+ '</span>',
590
+ ]
591
+ )
592
+
565
593
  out_diff: dict[Literal['text', 'markdown', 'html'], str] = {}
566
594
  command = directives['command']
567
- if (
568
- report_kind == 'html'
569
- and not command.startswith('wdiff')
570
- and _unfiltered_diff is not None
571
- and 'text' in _unfiltered_diff
572
- ):
573
- diff = _unfiltered_diff['text']
595
+ if report_kind == 'html' and _unfiltered_diff is not None and 'text' in _unfiltered_diff:
596
+ diff_text = ''.join(_unfiltered_diff['text'].splitlines(keepends=True)[2:])
574
597
  else:
575
598
  old_data = self.state.old_data
576
599
  new_data = self.state.new_data
@@ -606,12 +629,17 @@ class CommandDiffer(DifferBase):
606
629
  ) from subprocess.CalledProcessError(proc.returncode, cmdline)
607
630
  if proc.returncode == 0:
608
631
  self.state.verb = 'changed,no_report'
632
+ logger.info(
633
+ f"Job {self.job.index_number}: Command in differ 'command' returned 0 (no report) "
634
+ f'({self.job.get_location()})'
635
+ )
609
636
  return {'text': '', 'markdown': '', 'html': ''}
610
- head = '\n'.join(
637
+ head_text = '\n'.join(
611
638
  [
612
- f'Using differ "{directives}"',
613
- f'--- @ {self.make_timestamp(self.state.old_timestamp, tz)}',
614
- f'+++ @ {self.make_timestamp(self.state.new_timestamp, tz)}',
639
+ # f"Using command differ: {directives['command']}",
640
+ f'\033[91m--- @ {self.make_timestamp(self.state.old_timestamp, tz)}\033[0m',
641
+ f'\033[92m+++ @ {self.make_timestamp(self.state.new_timestamp, tz)}\033[0m',
642
+ '',
615
643
  ]
616
644
  )
617
645
  diff = proc.stdout
@@ -625,20 +653,33 @@ class CommandDiffer(DifferBase):
625
653
  if any(x in line for x in {'{+', '+}', '[-', '-]'}):
626
654
  keeplines.append(line)
627
655
  diff = ''.join(keeplines)
628
- diff = f'{head}\n{diff}'
629
- out_diff.update(
630
- {
631
- 'text': diff,
632
- 'markdown': diff,
633
- }
634
- )
656
+ if directives.get('is_html'):
657
+ diff_text = self.html2text(diff)
658
+ out_diff.update(
659
+ {
660
+ 'text': head_text + diff_text,
661
+ 'markdown': head_text + diff_text,
662
+ 'html': head_html + diff,
663
+ }
664
+ )
665
+ else:
666
+ diff_text = diff
667
+ out_diff.update(
668
+ {
669
+ 'text': head_text + diff_text,
670
+ 'markdown': head_text + diff_text,
671
+ }
672
+ )
635
673
 
636
- if report_kind == 'html':
674
+ if report_kind == 'html' and 'html' not in out_diff:
637
675
  if command.startswith('wdiff'):
638
676
  # colorize output of wdiff
639
- out_diff['html'] = self.wdiff_to_html(diff)
677
+ out_diff['html'] = head_html + self.wdiff_to_html(diff_text)
640
678
  else:
641
- out_diff['html'] = html.escape(diff)
679
+ out_diff['html'] = head_html + html.escape(diff_text)
680
+
681
+ if self.job.monospace and 'html' in out_diff:
682
+ out_diff['html'] += '</span>'
642
683
 
643
684
  return out_diff
644
685
 
@@ -991,6 +1032,7 @@ class ImageDiffer(DifferBase):
991
1032
  def ai_google(
992
1033
  old_image: Image.Image,
993
1034
  new_image: Image.Image,
1035
+ diff_image: Image.Image,
994
1036
  directives: AiGoogleDirectives,
995
1037
  ) -> str:
996
1038
  """Summarize changes in image using Generative AI (ALPHA)."""
@@ -1073,10 +1115,6 @@ class ImageDiffer(DifferBase):
1073
1115
  }
1074
1116
  }
1075
1117
 
1076
- # Create a diff image
1077
- # diff_image = ImageChops.difference(old_image, new_image)
1078
- # diff_image.format = new_image.format
1079
-
1080
1118
  # upload to Google
1081
1119
  additional_parts: list[dict[str, dict[str, str]]] = []
1082
1120
  executor = ThreadPoolExecutor()
@@ -1085,7 +1123,7 @@ class ImageDiffer(DifferBase):
1085
1123
  (
1086
1124
  ('old image', old_image),
1087
1125
  ('new image', new_image),
1088
- # ('differences image', diff_image),
1126
+ ('differences image', diff_image),
1089
1127
  ),
1090
1128
  ):
1091
1129
  if 'error' not in additional_part:
@@ -1107,22 +1145,33 @@ class ImageDiffer(DifferBase):
1107
1145
  'focus on the most significant changes.'
1108
1146
  )
1109
1147
  model_prompt = (
1148
+ 'You are a skilled visual analyst tasked with analyzing two versions of an image and summarizing the '
1149
+ 'key differences between them. The audience for your summary is already familiar with the '
1150
+ "image's content, so you should focus only on the most significant differences.\n\n"
1110
1151
  '**Instructions:**\n\n'
1111
- f"1. Compare the new image {additional_parts[1]['file_data']['file_uri']} to the old image "
1112
- f"{additional_parts[0]['file_data']['file_uri']}, focusing only on major changes.\n"
1113
- '2. Summarize the meaning of the changes in a clear and concise manner. Be specific.\n'
1114
- '3. Use Markdown formatting to structure your summary effectively. Use headings, bullet points, and '
1115
- 'other Markdown elements as needed to enhance readability.\n'
1116
- '4. Restrict your analysis and summary to these two specific images. Do not introduce external '
1117
- 'information or assumptions. Do not introduce knowledge from images you may have already seen.\n'
1152
+ '1. Carefully examine the yellow areas in the image '
1153
+ f"{additional_parts[2]['file_data']['file_uri']}, identify the differences, and describe them.\n"
1154
+ f"2. Refer to the old version of the image {additional_parts[0]['file_data']['file_uri']} and the new "
1155
+ f" version {additional_parts[1]['file_data']['file_uri']}.\n"
1156
+ '3. You are only interested in those differences, such as additions, removals, or alterations, that '
1157
+ 'modify the intended message or interpretation.\n'
1158
+ '4. Summarize the identified differences, except those ignored, in a clear and concise manner, '
1159
+ 'explaining how the meaning has shifted or evolved in the new version compared to the old version only '
1160
+ 'when necessary. Be specific and provide examples to illustrate your points when needed.\n'
1161
+ '5. If there are only additions to the image, then summarize the additions.\n'
1162
+ '6. Use Markdown formatting to structure your summary effectively. Use headings, bullet points, '
1163
+ 'and other Markdown elements as needed to enhance readability.\n'
1164
+ '7. Restrict your analysis and summary to the information provided within these images. Do '
1165
+ 'not introduce external information or assumptions.\n'
1118
1166
  )
1119
- summary = AIGoogleDiffer._send_to_model(
1167
+ summary, _ = AIGoogleDiffer._send_to_model(
1120
1168
  self.job,
1121
1169
  system_instructions,
1122
1170
  model_prompt,
1123
1171
  additional_parts=additional_parts, # type: ignore[arg-type]
1124
1172
  directives=directives,
1125
1173
  )
1174
+
1126
1175
  return summary
1127
1176
 
1128
1177
  data_type = directives.get('data_type', 'url')
@@ -1134,8 +1183,8 @@ class ImageDiffer(DifferBase):
1134
1183
  if data_type == 'url':
1135
1184
  old_image = load_image_from_web(self.state.old_data)
1136
1185
  new_image = load_image_from_web(self.state.new_data)
1137
- old_data = f' (<a href="{self.state.old_data}">Old image</a>)'
1138
- new_data = f' (<a href="{self.state.new_data}">New image</a>)'
1186
+ old_data = f' (<a href="{self.state.old_data}" target="_blank">Old image</a>)'
1187
+ new_data = f' (<a href="{self.state.new_data}" target="_blank">New image</a>)'
1139
1188
  elif data_type == 'ascii85':
1140
1189
  old_image = load_image_from_ascii85(self.state.old_data)
1141
1190
  new_image = load_image_from_ascii85(self.state.new_data)
@@ -1149,8 +1198,8 @@ class ImageDiffer(DifferBase):
1149
1198
  else: # 'filename'
1150
1199
  old_image = load_image_from_file(self.state.old_data)
1151
1200
  new_image = load_image_from_file(self.state.new_data)
1152
- old_data = f' (<a href="file://{self.state.old_data}">Old image</a>)'
1153
- new_data = f' (<a href="file://{self.state.new_data}">New image</a>)'
1201
+ old_data = f' (<a href="file://{self.state.old_data}" target="_blank">Old image</a>)'
1202
+ new_data = f' (<a href="file://{self.state.new_data}" target="_blank">New image</a>)'
1154
1203
 
1155
1204
  # Check formats TODO: is it needed? under which circumstances?
1156
1205
  # if new_image.format != old_image.format:
@@ -1202,7 +1251,7 @@ class ImageDiffer(DifferBase):
1202
1251
  # prepare AI summary
1203
1252
  summary = ''
1204
1253
  if 'ai_google' in directives:
1205
- summary = ai_google(old_image, new_image, directives.get('ai_google', {}))
1254
+ summary = ai_google(old_image, new_image, diff_image, directives.get('ai_google', {}))
1206
1255
 
1207
1256
  # Prepare HTML output
1208
1257
  htm = [
@@ -1246,8 +1295,14 @@ class ImageDiffer(DifferBase):
1246
1295
  )
1247
1296
  footer = f'Summary generated by Google Generative AI (ai_google directive(s): {directives_text})'
1248
1297
  return {
1249
- 'text': f'{summary}\n\n\nA visualization is available in HTML reports.\n------------\n{footer}',
1250
- 'markdown': f'{summary}\n\n\nA visualization is available in HTML reports.\n* * *\n{footer}',
1298
+ 'text': (
1299
+ f'{summary}\n\n\nA visualization of differences is available in {__package__} HTML reports.'
1300
+ f'\n------------\n{footer}'
1301
+ ),
1302
+ 'markdown': (
1303
+ f'{summary}\n\n\nA visualization of differences is available in {__package__} HTML reports.'
1304
+ f'\n* * *\n{footer}'
1305
+ ),
1251
1306
  'html': '<br>\n'.join(
1252
1307
  [
1253
1308
  mark_to_html(summary, extras={'tables'}).replace('<h2>', '<h3>').replace('</h2>', '</h3>'),
@@ -1298,20 +1353,17 @@ class AIGoogleDiffer(DifferBase):
1298
1353
  model_prompt: str,
1299
1354
  additional_parts: list[dict[str, str | dict[str, str]]] | None = None,
1300
1355
  directives: AiGoogleDirectives | None = None,
1301
- ) -> str:
1302
- """Creates the summary request to the model"""
1356
+ ) -> tuple[str, str]:
1357
+ """Creates the summary request to the model; returns the summary and the version of the actual model used."""
1303
1358
  api_version = '1beta'
1304
1359
  if directives is None:
1305
1360
  directives = {}
1306
- if 'model' not in directives:
1307
- directives['model'] = 'gemini-2.0-flash' # also for footer
1308
- model = directives.get('model')
1361
+ model = directives.get('model', 'gemini-2.0-flash')
1309
1362
  timeout = directives.get('timeout', 300)
1310
1363
  max_output_tokens = directives.get('max_output_tokens')
1311
1364
  temperature = directives.get('temperature', 0.0)
1312
- top_p = directives.get('top_p')
1365
+ top_p = directives.get('top_p', 1.0 if temperature == 0.0 else None)
1313
1366
  top_k = directives.get('top_k')
1314
-
1315
1367
  GOOGLE_AI_API_KEY = os.environ.get('GOOGLE_AI_API_KEY', '').rstrip()
1316
1368
  if len(GOOGLE_AI_API_KEY) != 39:
1317
1369
  logger.error(
@@ -1321,7 +1373,8 @@ class AIGoogleDiffer(DifferBase):
1321
1373
  return (
1322
1374
  f'## ERROR in summarizing changes using Google AI:\n'
1323
1375
  f'Environment variable GOOGLE_AI_API_KEY not found or is of the incorrect length '
1324
- f'{len(GOOGLE_AI_API_KEY)}.\n'
1376
+ f'{len(GOOGLE_AI_API_KEY)}.\n',
1377
+ '',
1325
1378
  )
1326
1379
 
1327
1380
  data: dict[str, Any] = {
@@ -1339,6 +1392,7 @@ class AIGoogleDiffer(DifferBase):
1339
1392
  if directives.get('tools'):
1340
1393
  data['tools'] = directives['tools']
1341
1394
  logger.info(f'Job {job.index_number}: Making the content generation request to Google AI model {model}')
1395
+ model_version = model # default
1342
1396
  try:
1343
1397
  r = httpx.Client(http2=True).post( # noqa: S113 Call to httpx without timeout
1344
1398
  f'https://generativelanguage.googleapis.com/v{api_version}/models/{model}:generateContent?'
@@ -1358,6 +1412,8 @@ class AIGoogleDiffer(DifferBase):
1358
1412
  f'AI summary unavailable: Model did not return any candidate output:\n'
1359
1413
  f'{jsonlib.dumps(result, ensure_ascii=True, indent=2)}'
1360
1414
  )
1415
+ model_version = result['modelVersion']
1416
+
1361
1417
  elif r.status_code == 400:
1362
1418
  summary = (
1363
1419
  f'AI summary unavailable: Received error from {r.url.host}: '
@@ -1375,7 +1431,7 @@ class AIGoogleDiffer(DifferBase):
1375
1431
  f'AI summary unavailable: HTTP client error: {e} when requesting data from ' f'{e.request.url.host}'
1376
1432
  )
1377
1433
 
1378
- return summary
1434
+ return summary, model_version
1379
1435
 
1380
1436
  def differ(
1381
1437
  self,
@@ -1392,8 +1448,8 @@ class AIGoogleDiffer(DifferBase):
1392
1448
  RuntimeWarning,
1393
1449
  )
1394
1450
 
1395
- def get_ai_summary(prompt: str, system_instructions: str) -> str:
1396
- """Generate AI summary from unified diff, or an error message"""
1451
+ def get_ai_summary(prompt: str, system_instructions: str) -> tuple[str, str]:
1452
+ """Generate AI summary from unified diff, or an error message, plus the model version."""
1397
1453
  GOOGLE_AI_API_KEY = os.environ.get('GOOGLE_AI_API_KEY', '').rstrip()
1398
1454
  if len(GOOGLE_AI_API_KEY) != 39:
1399
1455
  logger.error(
@@ -1403,12 +1459,10 @@ class AIGoogleDiffer(DifferBase):
1403
1459
  return (
1404
1460
  f'## ERROR in summarizing changes using {self.__kind__}:\n'
1405
1461
  f'Environment variable GOOGLE_AI_API_KEY not found or is of the incorrect length '
1406
- f'{len(GOOGLE_AI_API_KEY)}.\n'
1462
+ f'{len(GOOGLE_AI_API_KEY)}.\n',
1463
+ '',
1407
1464
  )
1408
1465
 
1409
- if 'model' not in directives:
1410
- directives['model'] = 'gemini-1.5-flash-latest' # also for footer
1411
-
1412
1466
  if '{unified_diff' in prompt: # matches unified_diff or unified_diff_new
1413
1467
  default_context_lines = 9999 if '{unified_diff}' in prompt else 0 # none if only unified_diff_new
1414
1468
  context_lines = directives.get('prompt_ud_context_lines', default_context_lines)
@@ -1425,7 +1479,7 @@ class AIGoogleDiffer(DifferBase):
1425
1479
  )
1426
1480
  if not unified_diff:
1427
1481
  # no changes
1428
- return ''
1482
+ return '', ''
1429
1483
  else:
1430
1484
  unified_diff = ''
1431
1485
 
@@ -1440,7 +1494,7 @@ class AIGoogleDiffer(DifferBase):
1440
1494
 
1441
1495
  # check if data is different (same data is sent during testing)
1442
1496
  if '{old_text}' in prompt and '{new_text}' in prompt and self.state.old_data == self.state.new_data:
1443
- return ''
1497
+ return '', ''
1444
1498
 
1445
1499
  model_prompt = prompt.format(
1446
1500
  unified_diff=unified_diff,
@@ -1449,14 +1503,14 @@ class AIGoogleDiffer(DifferBase):
1449
1503
  new_text=self.state.new_data,
1450
1504
  )
1451
1505
 
1452
- summary = self._send_to_model(
1506
+ summary, model_version = self._send_to_model(
1453
1507
  self.job,
1454
1508
  system_instructions,
1455
1509
  model_prompt,
1456
1510
  directives=directives,
1457
1511
  )
1458
1512
 
1459
- return summary
1513
+ return summary, model_version
1460
1514
 
1461
1515
  if directives.get('additions_only') or self.job.additions_only:
1462
1516
  default_system_instructions = (
@@ -1472,7 +1526,7 @@ class AIGoogleDiffer(DifferBase):
1472
1526
  'You are a skilled journalist tasked with analyzing two versions of a text and summarizing the key '
1473
1527
  'differences in meaning between them. The audience for your summary is already familiar with the '
1474
1528
  "text's content, so you can focus on the most significant changes.\n\n"
1475
- "**Instructions:**\n\n'"
1529
+ '**Instructions:**\n\n'
1476
1530
  '1. Carefully examine the old version of the text, provided within the `<old_version>` and '
1477
1531
  '`</old_version>` tags.\n'
1478
1532
  '2. Carefully examine the new version of the text, provided within the `<new_version>` and '
@@ -1483,25 +1537,38 @@ class AIGoogleDiffer(DifferBase):
1483
1537
  '5. Summarize the identified differences, except those ignored, in a clear and concise manner, '
1484
1538
  'explaining how the meaning has shifted or evolved in the new version compared to the old version only '
1485
1539
  'when necessary. Be specific and provide examples to illustrate your points when needed.\n'
1486
- '6. If ther are only additions to the text, then summarize the additions.\n'
1540
+ '6. If there are only additions to the text, then summarize the additions.\n'
1487
1541
  '7. Use Markdown formatting to structure your summary effectively. Use headings, bullet points, '
1488
1542
  'and other Markdown elements as needed to enhance readability.\n'
1489
1543
  '8. Restrict your analysis and summary to the information provided within the `<old_version>` and '
1490
- '`<new_version> tags. Do not introduce external information or assumptions.\n'
1544
+ '`<new_version>` tags. Do not introduce external information or assumptions.\n'
1491
1545
  )
1492
1546
  default_prompt = '<old_version>\n{old_text}\n</old_version>\n\n<new_version>\n{new_text}\n</new_version>'
1493
1547
  system_instructions = directives.get('system_instructions', default_system_instructions)
1494
1548
  prompt = directives.get('prompt', default_prompt).replace('\\n', '\n')
1495
- summary = get_ai_summary(prompt, system_instructions)
1549
+ summary, model_version = get_ai_summary(prompt, system_instructions)
1496
1550
  if not summary:
1497
1551
  self.state.verb = 'changed,no_report'
1498
1552
  return {'text': '', 'markdown': '', 'html': ''}
1499
1553
  newline = '\n' # For Python < 3.12 f-string {} compatibility
1500
1554
  back_n = '\\n' # For Python < 3.12 f-string {} compatibility
1501
- directives_text = (
1502
- ', '.join(f'{key}={str(value).replace(newline, back_n)}' for key, value in directives.items()) or 'None'
1555
+ directives.pop('model', None)
1556
+ if directives:
1557
+ directives_text = (
1558
+ ' (differ directive(s): '
1559
+ + (
1560
+ ', '.join(f'{key}={str(value).replace(newline, back_n)}' for key, value in directives.items())
1561
+ or 'None'
1562
+ )
1563
+ + ')'
1564
+ )
1565
+ else:
1566
+ directives_text = ''
1567
+ footer = (
1568
+ f"Summary by Google Generative AI's model {model_version}{directives_text}"
1569
+ if model_version and directives_text
1570
+ else ''
1503
1571
  )
1504
- footer = f'Summary generated by Google Generative AI (differ directive(s): {directives_text})'
1505
1572
  temp_unfiltered_diff: dict[Literal['text', 'markdown', 'html'], str] = {}
1506
1573
  for rep_kind in ['text', 'html']: # markdown is same as text
1507
1574
  unified_report = DifferBase.process(
@@ -1513,17 +1580,16 @@ class AIGoogleDiffer(DifferBase):
1513
1580
  temp_unfiltered_diff,
1514
1581
  )
1515
1582
  return {
1516
- 'text': f"{summary}\n\n{unified_report['text']}\n------------\n{footer}",
1517
- 'markdown': f"{summary}\n\n{unified_report['markdown']}\n* * *\n{footer}",
1583
+ 'text': f"{summary}\n\n{unified_report['text']}" + (f'\n------------\n{footer}' if footer else ''),
1584
+ 'markdown': f"{summary}\n\n{unified_report['markdown']}" + (f'\n* * *\n{footer}' if footer else ''),
1518
1585
  'html': '\n'.join(
1519
1586
  [
1520
1587
  mark_to_html(summary, extras={'tables'}).replace('<h2>', '<h3>').replace('</h2>', '</h3>'),
1521
1588
  '<br>',
1522
1589
  '<br>',
1523
1590
  unified_report['html'],
1524
- '-----<br>',
1525
- f'<i><small>{footer}</small></i>',
1526
1591
  ]
1592
+ + (['-----<br>', f'<i><small>{footer}</small></i>'] if footer else [])
1527
1593
  ),
1528
1594
  }
1529
1595
 
@@ -1575,10 +1641,13 @@ class WdiffDiffer(DifferBase):
1575
1641
  add_html = '<span style="background-color:#d1ffd1;color:#082b08;">'
1576
1642
  rem_html = '<span style="background-color:#fff0f0;color:#9c1c1c;text-decoration:line-through;">'
1577
1643
 
1578
- head_text = (
1579
- # f'Differ: wdiff\n'
1580
- f'\033[91m--- @ {self.make_timestamp(self.state.old_timestamp, tz)}\033[0m\n'
1581
- f'\033[92m+++ @ {self.make_timestamp(self.state.new_timestamp, tz)}\033[0m\n'
1644
+ head_text = '\n'.join(
1645
+ [
1646
+ # f'Differ: wdiff',
1647
+ f'\033[91m--- @ {self.make_timestamp(self.state.old_timestamp, tz)}\033[0m',
1648
+ f'\033[92m+++ @ {self.make_timestamp(self.state.new_timestamp, tz)}\033[0m',
1649
+ '',
1650
+ ]
1582
1651
  )
1583
1652
  head_html = '<br>\n'.join(
1584
1653
  [
@@ -194,7 +194,7 @@ class FilterBase(metaclass=TrackSubClasses):
194
194
  if unknown_keys and '<any>' not in allowed_keys:
195
195
  raise ValueError(
196
196
  f'Job {job_index_number}: Filter {filter_kind} does not support subfilter or filter '
197
- f"directive(s) {', '.join(unknown_keys)} (supported: {', '.join(allowed_keys)})."
197
+ f"directive(s) {', '.join(unknown_keys)}. Only {', '.join(allowed_keys)} are supported."
198
198
  )
199
199
 
200
200
  yield filter_kind, subfilter
@@ -794,7 +794,10 @@ class FormatJsonFilter(FilterBase):
794
794
  self.job.set_to_monospace()
795
795
  sort_keys = subfilter.get('sort_keys', False)
796
796
  indentation = int(subfilter.get('indentation', 4))
797
- parsed_json = jsonlib.loads(data)
797
+ try:
798
+ parsed_json = jsonlib.loads(data)
799
+ except jsonlib.JSONDecodeError as e:
800
+ return f"Filter '{self.__kind__}' returned JSONDecodeError: {e}\n\n{data!s}", mime_type
798
801
  if not mime_type.endswith('json'):
799
802
  mime_type = 'application/json'
800
803
  return jsonlib.dumps(parsed_json, ensure_ascii=False, sort_keys=sort_keys, indent=indentation), mime_type
@@ -67,7 +67,7 @@ class JobState(ContextManager):
67
67
  exception: Exception | None = None
68
68
  generated_diff: dict[Literal['text', 'markdown', 'html'], str]
69
69
  history_dic_snapshots: dict[str | bytes, Snapshot]
70
- new_data: str | bytes
70
+ new_data: str | bytes = ''
71
71
  new_error_data: ErrorData = {}
72
72
  new_etag: str
73
73
  new_mime_type: str = ''
@@ -234,18 +234,6 @@ class JobState(ContextManager):
234
234
  f'{dict(data=data, etag=self.new_etag, mime_type=mime_type)}'
235
235
  )
236
236
 
237
- # Apply automatic filters first
238
- filtered_data, mime_type = FilterBase.auto_process(self, data, mime_type)
239
-
240
- # Apply any specified filters
241
- for filter_kind, subfilter in FilterBase.normalize_filter_list(self.job.filters, self.job.index_number):
242
- filtered_data, mime_type = FilterBase.process(
243
- filter_kind, subfilter, self, filtered_data, mime_type
244
- )
245
-
246
- self.new_data = filtered_data
247
- self.new_mime_type = mime_type
248
-
249
237
  except Exception as e:
250
238
  # Job has a chance to format and ignore its error
251
239
  if self.debugger_attached():
@@ -258,31 +246,45 @@ class JobState(ContextManager):
258
246
  if not (self.error_ignored or isinstance(e, NotModifiedError)):
259
247
  self.tries += 1
260
248
  self.new_error_data = {
261
- 'type': type(self.exception).__name__,
262
- 'message': str(self.exception),
249
+ 'type': e.__class__.__name__,
250
+ 'message': str(e),
263
251
  }
264
252
  logger.info(
265
253
  f'Job {self.job.index_number}: Job ended with error; incrementing cumulative error runs to '
266
254
  f'{self.tries}'
267
255
  )
256
+
257
+ else:
258
+ # Apply automatic filters first
259
+ filtered_data, mime_type = FilterBase.auto_process(self, data, mime_type)
260
+
261
+ # Apply any specified filters
262
+ for filter_kind, subfilter in FilterBase.normalize_filter_list(self.job.filters, self.job.index_number):
263
+ filtered_data, mime_type = FilterBase.process(
264
+ filter_kind, subfilter, self, filtered_data, mime_type
265
+ )
266
+
267
+ self.new_data = filtered_data
268
+ self.new_mime_type = mime_type
269
+
268
270
  except Exception as e:
269
- # Job failed its chance to handle error
271
+ # Processing error or job failed its chance to handle error
270
272
  if self.debugger_attached():
271
273
  logger.warning('Running in a debugger: raising the exception instead of processing it')
272
274
  raise
275
+ self.new_timestamp = time.time()
273
276
  self.exception = e
274
- self.traceback = self.job.format_error(e, traceback.format_exc())
277
+ self.traceback = ''.join(traceback.format_exception_only(e, show_group=True)).rstrip()
275
278
  self.error_ignored = False
276
- if not isinstance(e, NotModifiedError):
277
- self.tries += 1
278
- self.new_error_data = {
279
- 'type': type(self.exception).__name__,
280
- 'message': str(self.exception),
281
- }
282
- logger.info(
283
- f'Job {self.job.index_number}: Job ended with error (internal handling failed); incrementing '
284
- f'cumulative error runs to {self.tries}'
285
- )
279
+ self.tries += 1
280
+ self.new_error_data = {
281
+ 'type': '.'.join(filter(None, [getattr(e, '__module__', None), e.__class__.__name__])),
282
+ 'message': str(e),
283
+ }
284
+ logger.info(
285
+ f'Job {self.job.index_number}: Job ended with error (internal handling failed); incrementing '
286
+ f'cumulative error runs to {self.tries}'
287
+ )
286
288
 
287
289
  logger.debug(f'Job {self.job.index_number}: Processed as {self.added_data()}')
288
290
  logger.info(f'{self.job.get_indexed_location()} ended processing')
@@ -323,7 +325,7 @@ class JobState(ContextManager):
323
325
  _generated_diff, _mime_type = FilterBase.process( # type: ignore[assignment]
324
326
  filter_kind, subfilter, self, _generated_diff, _mime_type
325
327
  )
326
- self.generated_diff[report_kind] = _generated_diff
328
+ self.generated_diff[report_kind] = str(_generated_diff)
327
329
 
328
330
  return self.generated_diff[report_kind]
329
331
 
@@ -786,7 +786,6 @@ class UrlJob(UrlJobBase):
786
786
  http_error_msg = f'{response.status_code} Client Error: {reason} for url: {response.url}'
787
787
  else:
788
788
  http_error_msg = f'{response.status_code} Server Error: {reason} for url: {response.url}'
789
- logger.error(f'httpx received {http_error_msg}')
790
789
 
791
790
  if response.status_code != 404:
792
791
  try:
@@ -858,8 +857,7 @@ class UrlJob(UrlJobBase):
858
857
  urllib3.util.ssl_.DEFAULT_CIPHERS += 'HIGH:!DH:!aNULL' # type: ignore[attr-defined]
859
858
  except AttributeError:
860
859
  logger.error(
861
- 'Unable to ignore_dh_key_too_small due to bug in '
862
- 'requests.packages.urrlib3.util.ssl.DEFAULT_CIPHERS'
860
+ 'Unable to ignore_dh_key_too_small due to bug in requests.packages.urrlib3.util.ssl.DEFAULT_CIPHERS'
863
861
  )
864
862
  logger.error('See https://github.com/psf/requests/issues/6443')
865
863
 
@@ -1105,7 +1103,6 @@ class UrlJob(UrlJobBase):
1105
1103
  ):
1106
1104
  # Instead of a full traceback, just show the error
1107
1105
  exception_str = str(exception).strip()
1108
- print(f'{exception_str=} {exception.args=} {type(exception)=}')
1109
1106
  if self.proxy and (
1110
1107
  (httpx and isinstance(exception, httpx.TransportError))
1111
1108
  or any(
@@ -1373,7 +1370,7 @@ class BrowserJob(UrlJobBase):
1373
1370
  ignore_default_args=ignore_default_args,
1374
1371
  timeout=timeout,
1375
1372
  headless=headless,
1376
- proxy=proxy,
1373
+ proxy=proxy, # type: ignore[arg-type]
1377
1374
  )
1378
1375
  browser_version = browser.version
1379
1376
  user_agent = headers.pop(
@@ -1402,7 +1399,7 @@ class BrowserJob(UrlJobBase):
1402
1399
  args=args,
1403
1400
  ignore_default_args=ignore_default_args,
1404
1401
  headless=headless,
1405
- proxy=proxy,
1402
+ proxy=proxy, # type: ignore[arg-type]
1406
1403
  no_viewport=no_viewport,
1407
1404
  ignore_https_errors=self.ignore_https_errors,
1408
1405
  extra_http_headers=dict(headers),
@@ -1811,7 +1808,7 @@ class BrowserJob(UrlJobBase):
1811
1808
  :param tb: The traceback.format_exc() string.
1812
1809
  :returns: A string to display and/or use in reports.
1813
1810
  """
1814
- exception_str = f'Browser error in {str(exception).strip()}'
1811
+ exception_str = str(exception).strip()
1815
1812
  print(f'{exception_str=}, {tb=}')
1816
1813
  if self.proxy and 'net::ERR' in exception_str:
1817
1814
  exception_str += f'\n\n(Job has proxy {self.proxy})'
@@ -117,7 +117,7 @@ if sys.platform == 'win32':
117
117
  try:
118
118
  from colorama import AnsiToWin32
119
119
  except ImportError as e: # pragma: no cover
120
- AnsiToWin32 = str(e) # type: ignore[assignment]
120
+ AnsiToWin32 = str(e) # type: ignore[assignment,misc]
121
121
 
122
122
  logger = logging.getLogger(__name__)
123
123
 
@@ -405,7 +405,7 @@ class HtmlReporter(ReporterBase):
405
405
  [mark_to_html(line, job_state.job.markdown_padded_tables) for line in text.splitlines()]
406
406
  )
407
407
 
408
- if job_state.verb == 'error':
408
+ if job_state.verb == 'error' or job_state.verb == 'repeated_error':
409
409
  htm = f'<pre style="white-space:pre-wrap;color:red;">{html.escape(job_state.traceback)}</pre>'
410
410
  if job_state.job.suppress_repeated_errors:
411
411
  htm += (
@@ -518,7 +518,7 @@ class TextReporter(ReporterBase):
518
518
  :param differ: The type of differ to use.
519
519
  :returns: HTML for a single job.
520
520
  """
521
- if job_state.verb == 'error':
521
+ if job_state.verb == 'error' or job_state.verb == 'repeated_error':
522
522
  if isinstance(self, StdoutReporter):
523
523
  text = self._red(job_state.traceback)
524
524
  else:
@@ -780,7 +780,7 @@ class MarkdownReporter(ReporterBase):
780
780
  :param differ: The type of differ to use.
781
781
  :returns: HTML for a single job.
782
782
  """
783
- if job_state.verb == 'error':
783
+ if job_state.verb == 'error' or job_state.verb == 'repeated_error':
784
784
  mark = job_state.traceback
785
785
  if job_state.job.suppress_repeated_errors:
786
786
  mark += '_Reminder: No further alerts until the error is resolved or changes._'
@@ -798,8 +798,8 @@ class YamlConfigStorage(BaseYamlFileStorage):
798
798
  )
799
799
  else:
800
800
  config['job_defaults']['command'] = config['job_defaults'].pop(
801
- 'shell'
802
- ) # type: ignore[typeddict-item]
801
+ 'shell' # type: ignore[typeddict-item]
802
+ )
803
803
 
804
804
  for key in {'all', 'url', 'browser', 'command'}:
805
805
  if key not in config['job_defaults']:
@@ -1,6 +1,6 @@
1
- Metadata-Version: 2.2
1
+ Metadata-Version: 2.4
2
2
  Name: webchanges
3
- Version: 3.28.1
3
+ Version: 3.29.0
4
4
  Summary: Web Changes Delivered. AI-Summarized. Totally Anonymous.
5
5
  Author-email: Mike Borsetti <mike+webchanges@borsetti.com>
6
6
  Maintainer-email: Mike Borsetti <mike+webchanges@borsetti.com>
@@ -163,6 +163,7 @@ Provides-Extra: safe-password
163
163
  Requires-Dist: keyring; extra == "safe-password"
164
164
  Provides-Extra: all
165
165
  Requires-Dist: webchanges[beautify,bs4,deepdiff_xml,html5lib,ical2text,imagediff,jq,matrix,ocr,pdf2text,pushbullet,pushover,pypdf_crypto,redis,requests,safe_password,use_browser,xmpp]; extra == "all"
166
+ Dynamic: license-file
166
167
 
167
168
  .. role:: underline
168
169
  :class: underline
@@ -195,7 +196,9 @@ For Generative AI summaries (BETA), you need a free `API Key from Google Cloud A
195
196
 
196
197
  Installation
197
198
  ============
198
- Install **webchanges** |pypi_version| |format| |status| |security| with:
199
+ |pypi_version| |format| |status| |security|
200
+
201
+ Install **webchanges** with:
199
202
 
200
203
  .. code-block:: bash
201
204
 
@@ -203,7 +206,7 @@ Install **webchanges** |pypi_version| |format| |status| |security| with:
203
206
 
204
207
  Running in Docker
205
208
  -----------------
206
- **webchanges** can easily run in a container; you can find a `Docker <https://www.docker.com/>`__ implementation
209
+ **webchanges** can easily run in a container and you will find a `Docker <https://www.docker.com/>`__ implementation
207
210
  `here <https://github.com/yubiuser/webchanges-docker>`__.
208
211
 
209
212
 
@@ -247,20 +250,19 @@ Schedule
247
250
  --------
248
251
  **webchanges** leverages the power of a system scheduler:
249
252
 
250
- - On Linux you can use cron, and a tool like `crontab.guru <https://crontab.guru>`__ can build a
251
- schedule expression for you (note: see `here <https://www.computerhope.com/unix/ucrontab.htm>`__ if you have never
252
- used cron before);
253
+ - On Linux you can use cron, with the help of a tool like `crontab.guru <https://crontab.guru>`__ (help `here
254
+ <https://www.computerhope.com/unix/ucrontab.htm>`__);
253
255
  - On Windows you can use `Windows Task Scheduler <https://en.wikipedia.org/wiki/Windows_Task_Scheduler>`__;
254
256
  - On macOS you can use `launchd <https://developer.apple
255
- .com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/ScheduledJobs.html>`__ (note: see `here
256
- <https://launchd.info/>`__ if you have never used launchd before).
257
+ .com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/ScheduledJobs.html>`__ (help `here
258
+ <https://launchd.info/>`__).
257
259
 
258
260
 
259
261
  Code
260
262
  ====
261
- |coveralls| |issues|
263
+ |coveralls| |issues| |code_style|
262
264
 
263
- The code and issues tracker are hosted on `GitHub <https://github.com/mborsetti/webchanges>`__.
265
+ The code, issues tracker, and discussions are hosted on `GitHub <https://github.com/mborsetti/webchanges>`__.
264
266
 
265
267
 
266
268
  Contributing
@@ -344,7 +346,7 @@ Example enhancements to HTML reporting:
344
346
  .. |format| image:: https://img.shields.io/pypi/format/webchanges.svg
345
347
  :target: https://pypi.org/project/webchanges/
346
348
  :alt: Kit format
347
- .. |downloads| image:: https://static.pepy.tech/badge/webchanges
349
+ .. |downloads| image:: https://img.shields.io/pypi/dm/webchanges.svg
348
350
  :target: https://www.pepy.tech/project/webchanges
349
351
  :alt: PyPI downloads
350
352
  .. |license| image:: https://img.shields.io/pypi/l/webchanges.svg
@@ -356,15 +358,24 @@ Example enhancements to HTML reporting:
356
358
  .. |readthedocs| image:: https://img.shields.io/readthedocs/webchanges/stable.svg?label=
357
359
  :target: https://webchanges.readthedocs.io/
358
360
  :alt: Documentation status
359
- .. |CI| image:: https://github.com/mborsetti/webchanges/actions/workflows/ci-cd.yaml/badge.svg?event=push
361
+ .. |old_CI| image:: https://github.com/mborsetti/webchanges/actions/workflows/ci-cd.yaml/badge.svg?event=push
362
+ :target: https://github.com/mborsetti/webchanges/actions
363
+ :alt: CI testing status
364
+ .. |CI| image:: https://img.shields.io/github/check-runs/mborsetti/webchanges/main
360
365
  :target: https://github.com/mborsetti/webchanges/actions
361
366
  :alt: CI testing status
362
- .. |coveralls| image:: https://coveralls.io/repos/github/mborsetti/webchanges/badge.svg?branch=main
367
+ .. |old_coveralls| image:: https://coveralls.io/repos/github/mborsetti/webchanges/badge.svg?branch=main
368
+ :target: https://coveralls.io/github/mborsetti/webchanges?branch=main
369
+ :alt: Code coverage by Coveralls
370
+ .. |coveralls| image:: https://img.shields.io/coverallsCoverage/github/mborsetti/webchanges.svg
363
371
  :target: https://coveralls.io/github/mborsetti/webchanges?branch=main
364
372
  :alt: Code coverage by Coveralls
373
+ .. |code_style| image:: https://img.shields.io/badge/code%20style-black-000000.svg
374
+ :target: https://github.com/psf/black
375
+ :alt: Code style black
365
376
  .. |status| image:: https://img.shields.io/pypi/status/webchanges.svg
366
377
  :target: https://pypi.org/project/webchanges/
367
378
  :alt: Package stability
368
- .. |security| image:: https://img.shields.io/badge/security-bandit-yellow.svg
379
+ .. |security| image:: https://img.shields.io/badge/security-bandit-green.svg
369
380
  :target: https://github.com/PyCQA/bandit
370
381
  :alt: Security Status
@@ -4,6 +4,7 @@ README.rst
4
4
  pyproject.toml
5
5
  requirements.txt
6
6
  webchanges/__init__.py
7
+ webchanges/__main__.py
7
8
  webchanges/cli.py
8
9
  webchanges/command.py
9
10
  webchanges/config.py
File without changes
File without changes
File without changes
File without changes