pywaybackup 4.2.1__tar.gz → 4.2.2__tar.gz

{pywaybackup-4.2.1/pywaybackup.egg-info → pywaybackup-4.2.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: pywaybackup
-Version: 4.2.1
+Version: 4.2.2
 Summary: Query and download archive.org as simple as possible.
 Author-email: bitdruid <bitdruid@outlook.com>
 License: MIT License
@@ -75,6 +75,16 @@ Prebuilt executables for Windows, Linux and macOS are attached to each [release]
 - Run from a terminal with arguments like the pip version: `waybackup -h`
 - Or start it without arguments (e.g. double-click on Windows) to enter **interactive mode** — the tool will prompt you for URL, mode and optional settings.
 
+On Linux/macOS the downloaded file has to be made executable first:
+
+```bash
+curl -L -o waybackup https://github.com/bitdruid/python-wayback-machine-downloader/releases/latest/download/waybackup-linux
+chmod +x waybackup
+./waybackup -h
+```
+
+Move it to a directory in your `PATH` (e.g. `~/.local/bin`) to call it as `waybackup` from anywhere.
+
 ### Manual
 
 1. Clone the repository <br>
@@ -100,6 +110,8 @@ You can import pywaybackup into your own scripts and run it. Args are the same a
 Additional args:
 - `silent` (default False): If True, suppresses all output to the console.
 - `debug` (default True): If False, disables writing errors to the error log file.
+- `progress_callback` (default None): A function receiving the `status()` dict on every progress update.
+- `progress_interval` (default 5): Seconds between callback updates while snapshots are downloading.
 
 Use:
 - `run()`
@@ -160,6 +172,39 @@ output:
 }
 ```
 
+Instead of polling `status()`, pass a callback to receive the `status` dict every `progress_interval` seconds (works on `silent` and `progress` disabled).
+
+```python
+from pywaybackup import PyWayBackup
+
+def on_progress(status):
+    print(status)
+
+backup = PyWayBackup(
+  url="https://example.com",
+  last=True,
+  silent=True,
+  progress=False,
+  progress_callback=on_progress,
+  progress_interval=5
+)
+
+backup.run()
+```
+output:
+```bash
+{'task': 'downloading cdx', 'current': 0, 'total': 0, 'progress': '0'}
+{'task': 'preparing snapshots', 'current': 0, 'total': 0, 'progress': '0'}
+{'task': 'downloading snapshots', 'current': 15, 'total': 84, 'progress': '18%'}
+{'task': 'downloading snapshots', 'current': 54, 'total': 84, 'progress': '64%'}
+{'task': 'downloading snapshots', 'current': 84, 'total': 84, 'progress': '100%'}
+{'task': 'done', 'current': 84, 'total': 84, 'progress': '100%'}
+```
+
+Any callable taking one argument works - a function, a bound method, or a lambda for something short.
+
+An exception raised inside the callback is logged and ignored - it will not abort a running job. When using `run(daemon=True)`, the callback runs inside the spawned process, so its side effects are not visible to the parent.
+
 ## cli
 
 - `-h`, `--help`: Show the help message and exit. Version info is shown in the help header.
@@ -245,6 +290,9 @@ Parameters will change the download behavior for snapshots.
 - **`--no-redirect`**:<br>
   Disables following redirects of snapshots. Can prevent timestamp-folder mismatches caused by redirects.
 
+- **`--no-merge-www`**:<br>
+  Keeps `www.example.com` and `example.com` in separate folders. By default both are treated as the same site and written into one folder, as archive.org returns them mixed together for a single query. Only use this if the two hosts served genuinely different content.
+
 - **`--retry`** `<attempts>`:<br>
   Retry attempts for failed downloads.
 

{pywaybackup-4.2.1 → pywaybackup-4.2.2}/README.md

@@ -38,6 +38,16 @@ Prebuilt executables for Windows, Linux and macOS are attached to each [release]
 - Run from a terminal with arguments like the pip version: `waybackup -h`
 - Or start it without arguments (e.g. double-click on Windows) to enter **interactive mode** — the tool will prompt you for URL, mode and optional settings.
 
+On Linux/macOS the downloaded file has to be made executable first:
+
+```bash
+curl -L -o waybackup https://github.com/bitdruid/python-wayback-machine-downloader/releases/latest/download/waybackup-linux
+chmod +x waybackup
+./waybackup -h
+```
+
+Move it to a directory in your `PATH` (e.g. `~/.local/bin`) to call it as `waybackup` from anywhere.
+
 ### Manual
 
 1. Clone the repository <br>
@@ -63,6 +73,8 @@ You can import pywaybackup into your own scripts and run it. Args are the same a
 Additional args:
 - `silent` (default False): If True, suppresses all output to the console.
 - `debug` (default True): If False, disables writing errors to the error log file.
+- `progress_callback` (default None): A function receiving the `status()` dict on every progress update.
+- `progress_interval` (default 5): Seconds between callback updates while snapshots are downloading.
 
 Use:
 - `run()`
@@ -123,6 +135,39 @@ output:
 }
 ```
 
+Instead of polling `status()`, pass a callback to receive the `status` dict every `progress_interval` seconds (works on `silent` and `progress` disabled).
+
+```python
+from pywaybackup import PyWayBackup
+
+def on_progress(status):
+    print(status)
+
+backup = PyWayBackup(
+  url="https://example.com",
+  last=True,
+  silent=True,
+  progress=False,
+  progress_callback=on_progress,
+  progress_interval=5
+)
+
+backup.run()
+```
+output:
+```bash
+{'task': 'downloading cdx', 'current': 0, 'total': 0, 'progress': '0'}
+{'task': 'preparing snapshots', 'current': 0, 'total': 0, 'progress': '0'}
+{'task': 'downloading snapshots', 'current': 15, 'total': 84, 'progress': '18%'}
+{'task': 'downloading snapshots', 'current': 54, 'total': 84, 'progress': '64%'}
+{'task': 'downloading snapshots', 'current': 84, 'total': 84, 'progress': '100%'}
+{'task': 'done', 'current': 84, 'total': 84, 'progress': '100%'}
+```
+
+Any callable taking one argument works - a function, a bound method, or a lambda for something short.
+
+An exception raised inside the callback is logged and ignored - it will not abort a running job. When using `run(daemon=True)`, the callback runs inside the spawned process, so its side effects are not visible to the parent.
+
 ## cli
 
 - `-h`, `--help`: Show the help message and exit. Version info is shown in the help header.
@@ -208,6 +253,9 @@ Parameters will change the download behavior for snapshots.
 - **`--no-redirect`**:<br>
   Disables following redirects of snapshots. Can prevent timestamp-folder mismatches caused by redirects.
 
+- **`--no-merge-www`**:<br>
+  Keeps `www.example.com` and `example.com` in separate folders. By default both are treated as the same site and written into one folder, as archive.org returns them mixed together for a single query. Only use this if the two hosts served genuinely different content.
+
 - **`--retry`** `<attempts>`:<br>
   Retry attempts for failed downloads.
 

{pywaybackup-4.2.1 → pywaybackup-4.2.2}/pyproject.toml

@@ -16,7 +16,7 @@ dependencies = [
     "python-magic-standalone==0.4.28",
 ]
 name = "pywaybackup"
-version = "4.2.1"
+version = "4.2.2"
 description = "Query and download archive.org as simple as possible."
 authors = [{ name = "bitdruid", email = "bitdruid@outlook.com" }]
 license = { file = "LICENSE" }
@@ -34,4 +34,4 @@ homepage = "https://github.com/bitdruid/python-wayback-machine-downloader"
 
 [tool.ruff]
 line-length = 120
-exclude = ["pywaybackup/Arguments.py"]
+exclude = ["pywaybackup/arguments.py"]

{pywaybackup-4.2.1 → pywaybackup-4.2.2}/pywaybackup/PyWayBackup.py

@@ -2,16 +2,17 @@ import multiprocessing
 import os
 import signal
 import sys
+import threading
 import time
 from importlib.metadata import version
-from typing import Union
+from typing import Callable, Optional, Union
 
 import pywaybackup.archive_save as archive_save
 from pywaybackup.archive_download import DownloadArchive
 from pywaybackup.db import Database as db
 from pywaybackup.Exception import Exception as ex
 from pywaybackup.files import CDXfile, CDXquery, CSVfile
-from pywaybackup.helper import sanitize_filename, url_split
+from pywaybackup.helper import normalize_domain, sanitize_filename, url_split
 from pywaybackup.SnapshotCollection import SnapshotCollection
 from pywaybackup.Verbosity import Verbosity as vb
 
@@ -89,6 +90,7 @@ class PyWayBackup:
         log (bool): Enable writing logs to a file.
         progress (bool): Show a progress bar.
         no_redirect (bool): Disable handling redirects.
+        no_merge_www (bool): Keep www and non-www snapshots in separate folders instead of merging them.
         retry (int): Retry attempts for failed downloads.
         workers (int): Number of download workers (default: 1).
         delay (int): Delay between download requests in seconds.
@@ -96,6 +98,9 @@ class PyWayBackup:
         keep (bool): Retain all job metadata after completion.
         silent (bool): Suppress all output (for programmatic use).
         debug (bool): Enable debug mode.
+        progress_callback (callable): Called with the status() dict on every task change and, during the
+            download phase, every `progress_interval` seconds. Fires independently of `silent`/`progress`.
+        progress_interval (float): Seconds between callback ticks while downloading snapshots (default: 5).
         **kwargs: Catch-all for future expansion or external integration.
 
     Methods:
@@ -127,6 +132,7 @@ class PyWayBackup:
         log: bool = False,
         progress: bool = False,
         no_redirect: bool = False,
+        no_merge_www: bool = False,
         retry: int = 0,
         workers: int = 1,
         delay: int = 0,
@@ -135,6 +141,8 @@ class PyWayBackup:
         keep: bool = False,
         silent: bool = True,
         debug: bool = False,
+        progress_callback: Optional[Callable[[dict], None]] = None,
+        progress_interval: float = 5.0,
         **kwargs: dict,
     ):
         self._url = url
@@ -155,6 +163,7 @@ class PyWayBackup:
         self._log = log
         self._progress = progress
         self._no_redirect = no_redirect
+        self._merge_www = not no_merge_www
         self._retry = retry
         self._workers = workers
         self._delay = delay
@@ -166,6 +175,8 @@ class PyWayBackup:
         # module exclusive
         self._silent = silent
         self._debug = debug
+        self._progress_callback = progress_callback
+        self._progress_interval = progress_interval
 
         # internal
         self._status = _Status()
@@ -328,7 +339,7 @@ class PyWayBackup:
             SnapshotCollection: The initialized and loaded snapshot collection.
         """
         collection = SnapshotCollection()
-        collection.load(mode=self._mode, cdxfile=self._cdxfile, csvfile=self._csvfile)
+        collection.load(mode=self._mode, cdxfile=self._cdxfile, csvfile=self._csvfile, merge_www=self._merge_www)
         collection.print_calculation()
         return collection
 
@@ -350,9 +361,43 @@ class PyWayBackup:
             delay=self._delay,
             wait=self._wait,
             workers=self._workers,
+            merge_www=self._merge_www,
         )
         downloader.run(SnapshotCollection=collection)
 
+    def _notify(self, task: str = None):
+        """
+        Set the current task (if given) and push the status to the progress callback.
+
+        Never gated by `silent`/`progress` - those only control the tqdm bar, while
+        embedders using the callback are exactly the ones running without one. A
+        failing callback is logged and swallowed, it must not abort a running job.
+
+        Args:
+            task (str, optional): New task name to set before notifying.
+        """
+        if task is not None:
+            self._status.task = task
+        if not self._progress_callback:
+            return
+        try:
+            self._progress_callback(self._status.status)
+        except Exception as e:
+            ex.exception(message="\nprogress_callback raised", e=e)
+
+    def _notify_loop(self, stop: threading.Event):
+        """
+        Push a status update every `progress_interval` seconds until stopped.
+
+        Runs in a daemon thread during the download phase, which can last hours
+        without any task change to notify on.
+
+        Args:
+            stop (threading.Event): Set by the workflow to end the loop.
+        """
+        while not stop.wait(self._progress_interval):
+            self._notify()
+
     def _workflow(self):
         """
         Executes the steps required to perform the backup process:
@@ -365,18 +410,23 @@ class PyWayBackup:
 
         """
         collection = None
+        ticker = None
+        ticker_stop = threading.Event()
         try:
             self._startup()
 
-            self._status.task = "downloading cdx"
+            self._notify(task="downloading cdx")
             cdx = self._prep_cdx()
 
             if cdx:
-                self._status.task = "preparing snapshots"
+                self._notify(task="preparing snapshots")
                 collection = self._prep_collection()
 
                 if collection:
-                    self._status.task = "downloading snapshots"
+                    self._notify(task="downloading snapshots")
+                    if self._progress_callback:
+                        ticker = threading.Thread(target=self._notify_loop, args=(ticker_stop,), daemon=True)
+                        ticker.start()
                     self._dl_download(collection=collection)
 
         except KeyboardInterrupt:
@@ -386,6 +436,10 @@ class PyWayBackup:
             self._keep = True
             ex.exception(message="", e=e)
         finally:
+            ticker_stop.set()  # stop ticking before the db teardown below
+            if ticker:
+                ticker.join()
+            self._notify()  # last frame before "done" freezes the counts
             # if a collection was created during the workflow, close its DB session cleanly
             try:
                 if collection:
@@ -412,7 +466,7 @@ class PyWayBackup:
         ... }
         """
         files = {
-            "snapshots": os.path.join(self._output, self._domain),
+            "snapshots": os.path.join(self._output, normalize_domain(self._domain, merge_www=self._merge_www)),
             "cdxfile": self._cdxfile.filepath,
             "dbfile": self._dbfile,
             "csvfile": self._csvfile.filepath,
@@ -492,7 +546,7 @@ class PyWayBackup:
 
     def _startup(self):
         if db.query_exist:
-            self._status.task = "resuming"
+            self._notify(task="resuming")
             vb.write(
                 content=f"\nDOWNLOAD job exist - processed: {db.query_progress}\nResuming download... (to reset the job use '--reset')"
             )
@@ -506,7 +560,7 @@ class PyWayBackup:
                     time.sleep(1)
 
     def _shutdown(self):
-        self._status.task = "done"
+        self._notify(task="done")  # counts stay frozen at the last live update
         collection = SnapshotCollection()
         collection.close()
         self._csvfile.store_result()

{pywaybackup-4.2.1 → pywaybackup-4.2.2}/pywaybackup/Snapshot.py

@@ -2,7 +2,7 @@ import os
 import threading
 
 from pywaybackup.db import Database, select, update, waybackup_snapshots, and_
-from pywaybackup.helper import url_split
+from pywaybackup.helper import normalize_domain, url_split
 from pywaybackup.Verbosity import Verbosity as vb
 
 
@@ -22,7 +22,7 @@ class Snapshot:
 
     __sqlite_lock = threading.Lock()
 
-    def __init__(self, db: Database, output: str, mode: str):
+    def __init__(self, db: Database, output: str, mode: str, merge_www: bool = True):
         """
         Initialize a Snapshot instance and fetch its database row if available.
 
@@ -30,10 +30,12 @@ class Snapshot:
             db (Database): Database connection/session manager.
             output (str): Output directory for downloaded files.
             mode (str): Download mode ('first', 'last', or default).
+            merge_www (bool): Write www and non-www snapshots into the same folder.
         """
         self._db = db
         self.output = output
         self.mode = mode
+        self.merge_www = merge_www
 
         self._redirect_url = None
         self._redirect_timestamp = None
@@ -164,10 +166,14 @@ class Snapshot:
         If mode is 'first' or 'last', the path does not include the timestamp.
         Otherwise, the timestamp is included in the path.
 
+        The domain is normalized so that the www and non-www variants of a site
+        end up in the same folder instead of two separate ones.
+
         Returns:
             str: Absolute path to the output file for the snapshot.
         """
         domain, subdir, filename = url_split(self.url_archive.split("id_/")[1], index=True)
+        domain = normalize_domain(domain, merge_www=self.merge_www)
 
         if self.mode == "last" or self.mode == "first":
             download_dir = os.path.join(self.output, domain, subdir)

{pywaybackup-4.2.1 → pywaybackup-4.2.2}/pywaybackup/SnapshotCollection.py

@@ -28,6 +28,9 @@ class SnapshotCollection:
 
         self._snapshot_faulty = 0  # error while parsing cdx line
 
+        self._merge_www = True  # treat www and non-www as the same url
+
+        self._filter_mailto = 0  # mailto: links in the cdx results
         self._filter_duplicates = 0  # with identical url_archive
         self._filter_mode = 0  # all snapshots filtered by the MODE (last or first)
         self._filter_skip = 0  # content of the csv file
@@ -60,12 +63,13 @@ class SnapshotCollection:
         self.db.write_progress(self._snapshot_handled, self._snapshot_total)
         self.db.session.close()
 
-    def load(self, mode: str, cdxfile: CDXfile, csvfile: CSVfile):
+    def load(self, mode: str, cdxfile: CDXfile, csvfile: CSVfile, merge_www: bool = True):
         """
         Insert the content of the cdx and csv file into the snapshot table.
         """
         self.cdxfile = cdxfile
         self.csvfile = csvfile
+        self._merge_www = merge_www
         if mode == "first":
             self._mode_first = True
         if mode == "last":
@@ -114,6 +118,9 @@ class SnapshotCollection:
                 "statuscode": line[3],
                 "origin": line[4],
             }
+            # cdx results contain mailto: links, which are no downloadable resources
+            if line["origin"].lower().startswith("mailto"):
+                return None
             url_archive = f"https://web.archive.org/web/{line['timestamp']}id_/{line['origin']}"
             statuscode = line["statuscode"] if line["statuscode"] in ("301", "404") else None
             return {
@@ -189,10 +196,14 @@ class SnapshotCollection:
                         line = line.rsplit(",", 1)[0]
 
                     try:
-                        line_batch.append(__parse_line(line))
+                        parsed = __parse_line(line)
                     except json.decoder.JSONDecodeError:
                         self._snapshot_faulty += 1
                         continue
+                    if parsed is None:
+                        self._filter_mailto += 1
+                        continue
+                    line_batch.append(parsed)
 
                     if len(line_batch) >= line_batchsize:
                         total_inserted += _insert_batch_safe(line_batch=line_batch)
@@ -253,6 +264,11 @@ class SnapshotCollection:
 
         - MODE_LAST → keep only the latest snapshot (highest timestamp) per url_origin.
         - MODE_FIRST → keep only the earliest snapshot (lowest timestamp) per url_origin.
+
+        Grouped by the url as it maps to disk, not by the raw url_origin: the output
+        path drops the scheme and normalizes the domain, so `http://www.example.com/`
+        and `https://example.com/` are one and the same file. Grouping by the raw value
+        would download both and let the second silently overwrite the first.
         """
 
         def _filter_mode():
@@ -261,11 +277,15 @@ class SnapshotCollection:
                 ordering = (
                     waybackup_snapshots.timestamp.desc() if self._mode_last else waybackup_snapshots.timestamp.asc()
                 )
-                # assign row numbers per url_origin
+                # same url as far as the output path is concerned (see helper.normalize_domain)
+                url_key = func.replace(func.lower(waybackup_snapshots.url_origin), "https://", "http://")
+                if self._merge_www:
+                    url_key = func.replace(url_key, "http://www.", "http://")
+                # assign row numbers per url
                 rownum = (
                     func.row_number()
                     .over(
-                        partition_by=waybackup_snapshots.url_origin,
+                        partition_by=url_key,
                         order_by=ordering,
                     )
                     .label("rn")
@@ -377,6 +397,8 @@ class SnapshotCollection:
         vb.write(content="\nSnapshot calculation:")
         vb.write(content=f"-----> {'in CDX file'.ljust(18)}: {self._cdx_total:,}")
 
+        if self._filter_mailto > 0:
+            vb.write(content=f"-----> {'removed mailto'.ljust(18)}: {self._filter_mailto:,}")
         if self._filter_duplicates > 0:
             vb.write(content=f"-----> {'removed duplicates'.ljust(18)}: {self._filter_duplicates:,}")
         if self._filter_mode > 0:

{pywaybackup-4.2.1 → pywaybackup-4.2.2}/pywaybackup/Worker.py

@@ -11,10 +11,11 @@ class Worker:
     Worker buffers its messages in a Message object. Output has to be done with write() method.
     """
 
-    def __init__(self, id: int, output: str, mode: str):
+    def __init__(self, id: int, output: str, mode: str, merge_www: bool = True):
         self.id = id
         self.output = output
         self.mode = mode
+        self.merge_www = merge_www
         self.message = Message(self)
 
     def init(self):
@@ -43,7 +44,7 @@ class Worker:
                 pass
 
     def assign_snapshot(self, total_amount: int):
-        self.snapshot = Snapshot(self.db, output=self.output, mode=self.mode)
+        self.snapshot = Snapshot(self.db, output=self.output, mode=self.mode, merge_www=self.merge_www)
         self.total_amount = total_amount
         if not self.snapshot.counter:  # counter only if a row was fetched
             self.snapshot = None

{pywaybackup-4.2.1 → pywaybackup-4.2.2}/pywaybackup/archive_download.py

@@ -81,7 +81,7 @@ class DownloadArchive:
         sc (SnapshotCollection): The snapshot collection being processed.
     """
 
-    def __init__(self, mode: str, output: str, retry: int, no_redirect: bool, delay: int, wait: int, workers: int):
+    def __init__(self, mode: str, output: str, retry: int, no_redirect: bool, delay: int, wait: int, workers: int, merge_www: bool = True):
         """
         Initialize the download manager with configuration options.
 
@@ -92,9 +92,11 @@ class DownloadArchive:
             no_redirect (bool): Disable redirect handling if True.
             delay (int): Delay between downloads in seconds.
             workers (int): Number of worker threads.
+            merge_www (bool): Write www and non-www snapshots into the same folder.
         """
         self.mode = mode
         self.output = output
+        self.merge_www = merge_www
         self.retry = retry
         self.no_redirect = no_redirect
         self.delay = delay
@@ -127,7 +129,7 @@ class DownloadArchive:
 
         threads = []
         for i in range(self.workers):
-            worker = Worker(id=i + 1, output=self.output, mode=self.mode)
+            worker = Worker(id=i + 1, output=self.output, mode=self.mode, merge_www=self.merge_www)
             vb.write(verbose=True, content=f"\n-----> Starting Worker: {worker.id}")
             thread = threading.Thread(target=self._download_loop, args=(worker,), daemon=True)
             threads.append(thread)

pywaybackup-4.2.2/pywaybackup/arguments.py

@@ -0,0 +1,72 @@
+import argparse
+import sys
+
+from argparse import RawTextHelpFormatter
+
+from importlib.metadata import version
+
+
+def build_parser() -> argparse.ArgumentParser:
+    """
+    Build the argument parser. Single source of truth for all cli arguments.
+
+    Split from parsing so that Interactive can read the argument definitions
+    without argparse touching sys.argv. Interactive walks the argument groups
+    below as they are, so a new argument shows up there without further work.
+    """
+    parser = argparse.ArgumentParser(
+        description=f"<<< python-wayback-machine-downloader v{version('pywaybackup')} >>>\nby @bitdruid -> https://github.com/bitdruid",
+        formatter_class=RawTextHelpFormatter,
+    )
+
+    required = parser.add_argument_group("required (one exclusive)")
+    required.add_argument("-u", "--url", type=str, metavar="", help="url (with subdir/subdomain) to download")
+    exclusive_required = required.add_mutually_exclusive_group(required=True)
+    exclusive_required.add_argument("-a", "--all", action="store_true", help="download snapshots of all timestamps")
+    exclusive_required.add_argument("-l", "--last", action="store_true", help="download the last version of each file snapshot")
+    exclusive_required.add_argument("-f", "--first", action="store_true", help="download the first version of each file snapshot")
+    exclusive_required.add_argument("-s", "--save", action="store_true", help="save a page to the wayback machine")
+
+    optional = parser.add_argument_group("optional query parameters")
+    optional.add_argument("-e", "--explicit", action="store_true", help="search only for the explicit given url")
+    optional.add_argument("-r", "--range", type=int, metavar="", help="range in years to search")
+    optional.add_argument("--start", type=int, metavar="", help="start timestamp format: YYYYMMDDHHMMSS")
+    optional.add_argument("--end", type=int, metavar="", help="end timestamp format: YYYYMMDDHHMMSS")
+    optional.add_argument("--limit", type=int, nargs="?", const=True, metavar="int", help="limit the number of snapshots to download")
+    optional.add_argument("--filetype", type=str, metavar="", help="filetypes to download comma separated (js,css,...)")
+    optional.add_argument("--statuscode", type=str, metavar="", help="statuscodes to download comma separated (200,404,...)")
+
+    behavior = parser.add_argument_group("manipulate behavior")
+    behavior.add_argument("-o", "--output", type=str, metavar="", help="output for all files - defaults to current directory")
+    behavior.add_argument("-m", "--metadata", type=str, metavar="", help="change directory for db/cdx/csv/log files")
+    behavior.add_argument("-v", "--verbose", type=str, nargs="?", const="default", metavar="", help="verbosity level: low, default, high (default if flag set without value)")
+    behavior.add_argument("--log", action="store_true", help="save a log file into the output folder")
+    behavior.add_argument("--progress", action="store_true", help="show a progress bar")
+    behavior.add_argument("--no-redirect", action="store_true", help="do not follow redirects by archive.org")
+    behavior.add_argument("--no-merge-www", action="store_true", help="keep www and non-www snapshots in separate folders")
+    behavior.add_argument("--retry", type=int, default=0, metavar="", help="retry failed downloads (opt tries as int, else infinite)")
+    behavior.add_argument("--workers", type=int, default=1, metavar="", help="number of workers (simultaneous downloads)")
+    behavior.add_argument("--delay", type=int, default=0, metavar="", help="delay between each download in seconds")
+    behavior.add_argument("--wait", type=int, default=15, metavar="", help="seconds to wait before renewing connection after HTTP errors or snapshot download errors (default: 15)")
+
+    special = parser.add_argument_group("special")
+    special.add_argument("--reset", action="store_true", help="reset the job and ignore existing cdx/db/csv files")
+    special.add_argument("--keep", action="store_true", help="keep all files after the job finished")
+
+    return parser
+
+
+class Arguments:
+    def __init__(self):
+        parser = build_parser()
+
+        args = parser.parse_args(args=None if sys.argv[1:] else ["--help"])  # if no arguments are given, print help
+
+        args.silent = False
+        args.debug = True
+
+        self.args = args
+
+    def get_args(self) -> dict:
+        """Returns the parsed arguments as a dictionary."""
+        return vars(self.args)

{pywaybackup-4.2.1 → pywaybackup-4.2.2}/pywaybackup/helper.py

@@ -47,6 +47,27 @@ def url_get_timestamp(url):
     return timestamp
 
 
+def normalize_domain(domain: str, merge_www: bool = True) -> str:
+    """
+    Normalize a domain for use as a folder name.
+
+    The cdx api canonicalizes hosts, so a single query returns `example.com`,
+    `www.example.com`, `www.example.com.` and `www.EXAMPLE.com` mixed together.
+    Without normalizing these all become separate output folders for the same site.
+
+    The `www.` prefix is only stripped if a dot remains, so `www.com` stays intact.
+    Real subdomains (`blog.example.com`) are left alone.
+
+    Args:
+        domain (str): The domain to normalize.
+        merge_www (bool): If False, keeps the `www.` prefix (--no-merge-www).
+    """
+    domain = domain.lower().rstrip(".")
+    if merge_www and domain.startswith("www.") and "." in domain[4:]:
+        domain = domain[4:]
+    return domain
+
+
 def url_split(url, index=False):
     """
     Split a URL into domain, subdir, and filename.

{pywaybackup-4.2.1 → pywaybackup-4.2.2}/pywaybackup/interactive.py

@@ -3,13 +3,14 @@ Interactive mode: prompt the user for arguments instead of parsing sys.argv.
 
 Used when waybackup is launched without CLI arguments (e.g. double-clicking
 the Windows .exe). Produces the same dict shape as Arguments.get_args() so
-PyWayBackup(**args) works either way. Argument metadata is read from
-arg_specs.ARG_SPECS so flags only have to be declared in one place.
+PyWayBackup(**args) works either way. The arguments and their grouping are read
+from the argparse parser, so a new argument is offered here automatically - it
+only has to be added in arguments.py.
 """
 
 from importlib.metadata import version
 
-from pywaybackup.arg_specs import ARG_SPECS, EXCLUSIVE_GROUPS, default_args
+from pywaybackup.arguments import build_parser
 
 
 class Interactive:
@@ -17,28 +18,35 @@ class Interactive:
         print(f"<<< python-wayback-machine-downloader v{version('pywaybackup')} >>>")
         print("Interactive mode - press Ctrl+C to abort.\n")
 
-        args = default_args()
-
-        # 1. Required URL
-        url_spec = _spec_by_name("url")
-        args["url"] = self._prompt_required(url_spec.prompt or url_spec.help)
-
-        # 2. Required exclusive group(s) — pick exactly one member
-        for ex_name in EXCLUSIVE_GROUPS:
-            members = [s for s in ARG_SPECS if s.exclusive_group == ex_name]
-            choice = self._prompt_choice(
-                ex_name.capitalize(),
-                [(s.name, s.help) for s in members],
-            )
-            for s in members:
-                args[s.name] = s.name == choice
-
-        # 3. Advanced options
-        if self._prompt_yes_no("Configure advanced options?", default=False):
-            for spec in ARG_SPECS:
-                if not spec.advanced:
-                    continue
-                args[spec.name] = self._prompt_for(spec, args[spec.name])
+        parser = build_parser()
+        actions = [action for action in parser._actions if action.dest != "help"]
+        args = {action.dest: action.default for action in actions}
+
+        # 1. Required url
+        url = _action_by_dest(actions, "url")
+        args["url"] = self._prompt_required(url.help)
+
+        # 2. Required exclusive group(s) - pick exactly one member
+        exclusive = set()
+        for group in parser._mutually_exclusive_groups:
+            members = group._group_actions
+            exclusive.update(action.dest for action in members)
+            if not group.required:
+                continue
+            choice = self._prompt_choice("Mode", [(a.dest, a.help) for a in members])
+            for action in members:
+                args[action.dest] = action.dest == choice
+
+        # 3. Every other argument, offered group by group as defined in the parser
+        handled = exclusive | {"url"}
+        for group in parser._action_groups:
+            members = [a for a in group._group_actions if a.dest not in handled and a.dest != "help"]
+            if not members:
+                continue
+            if not self._prompt_yes_no(f"\nConfigure {group.title}?", default=False):
+                continue
+            for action in members:
+                args[action.dest] = self._prompt_for(action, args[action.dest])
 
         # internal flags (parity with Arguments.py)
         args["silent"] = False
@@ -50,11 +58,11 @@ class Interactive:
     def get_args(self) -> dict:
         return self.args
 
-    def _prompt_for(self, spec, current):
-        label = spec.prompt or spec.help
-        if spec.action == "store_true":
-            return self._prompt_yes_no(f"{label}", default=bool(current))
-        if spec.type is int:
+    def _prompt_for(self, action, current):
+        label = action.help
+        if action.nargs == 0:  # store_true
+            return self._prompt_yes_no(label, default=bool(current))
+        if action.type is int:
             if current is None:
                 return self._prompt_optional_int(label)
             return self._prompt_int(label, default=current)
@@ -137,8 +145,8 @@ class Interactive:
             print(f"  (please enter one of: {', '.join(sorted(valid))})")
 
 
-def _spec_by_name(name):
-    for s in ARG_SPECS:
-        if s.name == name:
-            return s
-    raise KeyError(name)
+def _action_by_dest(actions, dest):
+    for action in actions:
+        if action.dest == dest:
+            return action
+    raise KeyError(dest)

{pywaybackup-4.2.1 → pywaybackup-4.2.2}/pywaybackup/main.py

@@ -2,7 +2,7 @@ import signal
 import sys
 
 from pywaybackup import PyWayBackup
-from pywaybackup.arg_parser import Arguments
+from pywaybackup.arguments import Arguments
 from pywaybackup.interactive import Interactive
 
 

{pywaybackup-4.2.1 → pywaybackup-4.2.2/pywaybackup.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: pywaybackup
-Version: 4.2.1
+Version: 4.2.2
 Summary: Query and download archive.org as simple as possible.
 Author-email: bitdruid <bitdruid@outlook.com>
 License: MIT License
@@ -75,6 +75,16 @@ Prebuilt executables for Windows, Linux and macOS are attached to each [release]
 - Run from a terminal with arguments like the pip version: `waybackup -h`
 - Or start it without arguments (e.g. double-click on Windows) to enter **interactive mode** — the tool will prompt you for URL, mode and optional settings.
 
+On Linux/macOS the downloaded file has to be made executable first:
+
+```bash
+curl -L -o waybackup https://github.com/bitdruid/python-wayback-machine-downloader/releases/latest/download/waybackup-linux
+chmod +x waybackup
+./waybackup -h
+```
+
+Move it to a directory in your `PATH` (e.g. `~/.local/bin`) to call it as `waybackup` from anywhere.
+
 ### Manual
 
 1. Clone the repository <br>
@@ -100,6 +110,8 @@ You can import pywaybackup into your own scripts and run it. Args are the same a
 Additional args:
 - `silent` (default False): If True, suppresses all output to the console.
 - `debug` (default True): If False, disables writing errors to the error log file.
+- `progress_callback` (default None): A function receiving the `status()` dict on every progress update.
+- `progress_interval` (default 5): Seconds between callback updates while snapshots are downloading.
 
 Use:
 - `run()`
@@ -160,6 +172,39 @@ output:
 }
 ```
 
+Instead of polling `status()`, pass a callback to receive the `status` dict every `progress_interval` seconds (works on `silent` and `progress` disabled).
+
+```python
+from pywaybackup import PyWayBackup
+
+def on_progress(status):
+    print(status)
+
+backup = PyWayBackup(
+  url="https://example.com",
+  last=True,
+  silent=True,
+  progress=False,
+  progress_callback=on_progress,
+  progress_interval=5
+)
+
+backup.run()
+```
+output:
+```bash
+{'task': 'downloading cdx', 'current': 0, 'total': 0, 'progress': '0'}
+{'task': 'preparing snapshots', 'current': 0, 'total': 0, 'progress': '0'}
+{'task': 'downloading snapshots', 'current': 15, 'total': 84, 'progress': '18%'}
+{'task': 'downloading snapshots', 'current': 54, 'total': 84, 'progress': '64%'}
+{'task': 'downloading snapshots', 'current': 84, 'total': 84, 'progress': '100%'}
+{'task': 'done', 'current': 84, 'total': 84, 'progress': '100%'}
+```
+
+Any callable taking one argument works - a function, a bound method, or a lambda for something short.
+
+An exception raised inside the callback is logged and ignored - it will not abort a running job. When using `run(daemon=True)`, the callback runs inside the spawned process, so its side effects are not visible to the parent.
+
 ## cli
 
 - `-h`, `--help`: Show the help message and exit. Version info is shown in the help header.
@@ -245,6 +290,9 @@ Parameters will change the download behavior for snapshots.
 - **`--no-redirect`**:<br>
   Disables following redirects of snapshots. Can prevent timestamp-folder mismatches caused by redirects.
 
+- **`--no-merge-www`**:<br>
+  Keeps `www.example.com` and `example.com` in separate folders. By default both are treated as the same site and written into one folder, as archive.org returns them mixed together for a single query. Only use this if the two hosts served genuinely different content.
+
 - **`--retry`** `<attempts>`:<br>
   Retry attempts for failed downloads.
 

{pywaybackup-4.2.1 → pywaybackup-4.2.2}/pywaybackup.egg-info/SOURCES.txt

@@ -10,8 +10,7 @@ pywaybackup/Worker.py
 pywaybackup/__init__.py
 pywaybackup/archive_download.py
 pywaybackup/archive_save.py
-pywaybackup/arg_parser.py
-pywaybackup/arg_specs.py
+pywaybackup/arguments.py
 pywaybackup/db.py
 pywaybackup/files.py
 pywaybackup/helper.py

pywaybackup-4.2.1/pywaybackup/arg_parser.py

@@ -1,55 +0,0 @@
-import argparse
-import sys
-from argparse import RawTextHelpFormatter
-from importlib.metadata import version
-
-from pywaybackup.arg_specs import ARG_GROUPS, ARG_SPECS, EXCLUSIVE_GROUPS
-
-
-class Arguments:
-    def __init__(self):
-        parser = argparse.ArgumentParser(
-            description=f"<<< python-wayback-machine-downloader v{version('pywaybackup')} >>>\nby @bitdruid -> https://github.com/bitdruid",
-            formatter_class=RawTextHelpFormatter,
-        )
-
-        groups = {name: parser.add_argument_group(name) for name in ARG_GROUPS}
-
-        exclusive = {
-            ex_name: groups[ex_meta["parent_group"]].add_mutually_exclusive_group(required=ex_meta["required"])
-            for ex_name, ex_meta in EXCLUSIVE_GROUPS.items()
-        }
-
-        for spec in ARG_SPECS:
-            target = exclusive[spec.exclusive_group] if spec.exclusive_group else groups[spec.group]
-            target.add_argument(*spec.flags, **_argparse_kwargs(spec))
-
-        args = parser.parse_args(args=None if sys.argv[1:] else ["--help"])  # if no arguments are given, print help
-
-        args.silent = False
-        args.debug = True
-
-        self.args = args
-
-    def get_args(self) -> dict:
-        """Returns the parsed arguments as a dictionary."""
-        return vars(self.args)
-
-
-def _argparse_kwargs(spec) -> dict:
-    """Translate an ArgSpec into kwargs for argparse.add_argument()."""
-    kwargs = {"help": spec.help}
-    if spec.action == "store_true":
-        kwargs["action"] = "store_true"
-        kwargs["default"] = bool(spec.default)
-    elif spec.action == "optional_value":
-        kwargs["type"] = spec.type
-        kwargs["nargs"] = "?"
-        kwargs["const"] = spec.const
-        kwargs["metavar"] = spec.metavar
-        kwargs["default"] = spec.default
-    else:
-        kwargs["type"] = spec.type
-        kwargs["metavar"] = spec.metavar
-        kwargs["default"] = spec.default
-    return kwargs

pywaybackup-4.2.1/pywaybackup/arg_specs.py

@@ -1,257 +0,0 @@
-"""
-Single source of truth for waybackup CLI arguments.
-
-Both Arguments (argparse) and Interactive (input prompts) consume this list,
-so a new flag only needs to be added in one place. PyWayBackup.__init__ keeps
-its explicit signature (vscode autocomplete).
-"""
-
-from dataclasses import dataclass
-from typing import Any, List, Optional
-
-
-@dataclass
-class ArgSpec:
-    name: str  # internal key (matches PyWayBackup.__init__ kwarg)
-    flags: List[str]  # CLI flags, e.g. ["-u", "--url"]
-    group: str  # argparse group label
-    help: str  # CLI help text
-    prompt: Optional[str] = None  # interactive prompt label (None = skip in interactive)
-    type: Optional[type] = None  # str / int / None for store_true
-    default: Any = None
-    action: str = "store"  # "store" | "store_true" | "optional_value"
-    const: Any = None  # used when action="optional_value"
-    metavar: str = ""  # argparse metavar; ignored for store_true
-    exclusive_group: Optional[str] = None  # name of a mutex group below
-    advanced: bool = False  # show in interactive only when advanced opts enabled
-
-
-# Mutually exclusive groups, keyed by name. Specs join them via exclusive_group=...
-EXCLUSIVE_GROUPS = {
-    "mode": {"required": True, "parent_group": "required (one exclusive)"},
-}
-
-# argparse groups in display order
-ARG_GROUPS = [
-    "required (one exclusive)",
-    "optional query parameters",
-    "manipulate behavior",
-    "special",
-]
-
-
-ARG_SPECS: List[ArgSpec] = [
-    # required
-    ArgSpec(
-        name="url",
-        flags=["-u", "--url"],
-        group="required (one exclusive)",
-        help="url (with subdir/subdomain) to download",
-        prompt="URL to download (with subdir/subdomain)",
-        type=str,
-    ),
-    ArgSpec(
-        name="all",
-        flags=["-a", "--all"],
-        group="required (one exclusive)",
-        help="download snapshots of all timestamps",
-        action="store_true",
-        default=False,
-        exclusive_group="mode",
-    ),
-    ArgSpec(
-        name="last",
-        flags=["-l", "--last"],
-        group="required (one exclusive)",
-        help="download the last version of each file snapshot",
-        action="store_true",
-        default=False,
-        exclusive_group="mode",
-    ),
-    ArgSpec(
-        name="first",
-        flags=["-f", "--first"],
-        group="required (one exclusive)",
-        help="download the first version of each file snapshot",
-        action="store_true",
-        default=False,
-        exclusive_group="mode",
-    ),
-    ArgSpec(
-        name="save",
-        flags=["-s", "--save"],
-        group="required (one exclusive)",
-        help="save a page to the wayback machine",
-        action="store_true",
-        default=False,
-        exclusive_group="mode",
-    ),
-    # -------------------- optional query --------------------
-    ArgSpec(
-        name="explicit",
-        flags=["-e", "--explicit"],
-        group="optional query parameters",
-        help="search only for the explicit given url",
-        action="store_true",
-        default=False,
-    ),
-    ArgSpec(
-        name="range",
-        flags=["-r", "--range"],
-        group="optional query parameters",
-        help="range in years to search",
-        prompt="Range in years to search",
-        type=int,
-        advanced=True,
-    ),
-    ArgSpec(
-        name="start",
-        flags=["--start"],
-        group="optional query parameters",
-        help="start timestamp format: YYYYMMDDHHMMSS",
-        type=int,
-    ),
-    ArgSpec(
-        name="end",
-        flags=["--end"],
-        group="optional query parameters",
-        help="end timestamp format: YYYYMMDDHHMMSS",
-        type=int,
-    ),
-    ArgSpec(
-        name="limit",
-        flags=["--limit"],
-        group="optional query parameters",
-        help="limit the number of snapshots to download",
-        prompt="Limit number of snapshots",
-        type=int,
-        action="optional_value",
-        const=True,
-        metavar="int",
-        advanced=True,
-    ),
-    ArgSpec(
-        name="filetype",
-        flags=["--filetype"],
-        group="optional query parameters",
-        help="filetypes to download comma separated (js,css,...)",
-        type=str,
-    ),
-    ArgSpec(
-        name="statuscode",
-        flags=["--statuscode"],
-        group="optional query parameters",
-        help="statuscodes to download comma separated (200,404,...)",
-        type=str,
-    ),
-    # behavior
-    ArgSpec(
-        name="output",
-        flags=["-o", "--output"],
-        group="manipulate behavior",
-        help="output for all files - defaults to current directory",
-        prompt="Output directory",
-        type=str,
-        advanced=True,
-    ),
-    ArgSpec(
-        name="metadata",
-        flags=["-m", "--metadata"],
-        group="manipulate behavior",
-        help="change directory for db/cdx/csv/log files",
-        type=str,
-    ),
-    ArgSpec(
-        name="verbose",
-        flags=["-v", "--verbose"],
-        group="manipulate behavior",
-        help="verbosity level: low, default, high (default if flag set without value)",
-        type=str,
-        action="optional_value",
-        const="default",
-    ),
-    ArgSpec(
-        name="log",
-        flags=["--log"],
-        group="manipulate behavior",
-        help="save a log file into the output folder",
-        prompt="Save log file?",
-        action="store_true",
-        default=False,
-        advanced=True,
-    ),
-    ArgSpec(
-        name="progress",
-        flags=["--progress"],
-        group="manipulate behavior",
-        help="show a progress bar",
-        prompt="Show progress bar?",
-        action="store_true",
-        default=False,
-        advanced=True,
-    ),
-    ArgSpec(
-        name="no_redirect",
-        flags=["--no-redirect"],
-        group="manipulate behavior",
-        help="do not follow redirects by archive.org",
-        action="store_true",
-        default=False,
-    ),
-    ArgSpec(
-        name="retry",
-        flags=["--retry"],
-        group="manipulate behavior",
-        help="retry failed downloads (opt tries as int, else infinite)",
-        type=int,
-        default=0,
-    ),
-    ArgSpec(
-        name="workers",
-        flags=["--workers"],
-        group="manipulate behavior",
-        help="number of workers (simultaneous downloads)",
-        prompt="Workers (parallel downloads)",
-        type=int,
-        default=1,
-        advanced=True,
-    ),
-    ArgSpec(
-        name="delay",
-        flags=["--delay"],
-        group="manipulate behavior",
-        help="delay between each download in seconds",
-        type=int,
-        default=0,
-    ),
-    ArgSpec(
-        name="wait",
-        flags=["--wait"],
-        group="manipulate behavior",
-        help="seconds to wait before renewing connection after HTTP errors or snapshot download errors (default: 15)",
-        type=int,
-        default=15,
-    ),
-    # special
-    ArgSpec(
-        name="reset",
-        flags=["--reset"],
-        group="special",
-        help="reset the job and ignore existing cdx/db/csv files",
-        action="store_true",
-        default=False,
-    ),
-    ArgSpec(
-        name="keep",
-        flags=["--keep"],
-        group="special",
-        help="keep all files after the job finished",
-        action="store_true",
-        default=False,
-    ),
-]
-
-
-def default_args() -> dict:
-    """Return a dict of {name: default} for every spec — the canonical empty arg payload."""
-    return {spec.name: spec.default for spec in ARG_SPECS}