pywaybackup 4.1.5__tar.gz → 4.2.0__tar.gz

{pywaybackup-4.1.5/pywaybackup.egg-info → pywaybackup-4.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: pywaybackup
-Version: 4.1.5
+Version: 4.2.0
 Summary: Query and download archive.org as simple as possible.
 Author-email: bitdruid <bitdruid@outlook.com>
 License: MIT License
@@ -26,11 +26,11 @@ License: MIT License
         SOFTWARE.
         
 Project-URL: homepage, https://github.com/bitdruid/python-wayback-machine-downloader
-Requires-Python: <3.14,>=3.8
+Requires-Python: >=3.8
 Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: ruff
-Requires-Dist: SQLAlchemy==2.0.43
+Requires-Dist: SQLAlchemy==2.0.51
 Requires-Dist: requests==2.32.3
 Requires-Dist: tqdm==4.67.1
 Requires-Dist: python-magic-standalone==0.4.28
@@ -39,7 +39,7 @@ Requires-Dist: python-magic-standalone==0.4.28
 
 [![PyPI](https://img.shields.io/pypi/v/pywaybackup)](https://pypi.org/project/pywaybackup/)
 [![PyPI - Downloads](https://img.shields.io/pypi/dm/pywaybackup)](https://pypi.org/project/pywaybackup/)
-![Python Version](https://img.shields.io/badge/Python-3.8-blue)
+![Python Version](https://img.shields.io/badge/Python-3.8%2B-blue)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 
 Downloading archived web pages from the [Wayback Machine](https://archive.org/web/).
@@ -68,6 +68,13 @@ This tool allows you to download content from the Wayback Machine (archive.org).
 2. Run the tool <br>
    `waybackup -h`
 
+### Standalone binary
+
+Prebuilt executables for Windows, Linux and macOS are attached to each [release](https://github.com/bitdruid/python-wayback-machine-downloader/releases). No Python required.
+
+- Run from a terminal with arguments like the pip version: `waybackup -h`
+- Or start it without arguments (e.g. double-click on Windows) to enter **interactive mode** — the tool will prompt you for URL, mode and optional settings.
+
 ### Manual
 
 1. Clone the repository <br>
@@ -155,8 +162,9 @@ output:
 
 ## cli
 
-- `-h`, `--help`: Show the help message and exit.
-- `-v`, `--version`: Show information about the tool and exit.
+- `-h`, `--help`: Show the help message and exit. Version info is shown in the help header.
+
+> **Interactive mode:** running `waybackup` without any arguments in a terminal starts a guided prompt for URL, mode and optional settings. Without a terminal (scripts/cron), the help is printed instead.
 
 #### Required
 
@@ -171,6 +179,8 @@ output:
   Last Version. Gives one folder containing the last version of each file of specified `--range`.
 - **`-f`**, **`--first`**:<br>
   First Version. Gives one folder containing the first version of each file of specified `--range`.
+- **`-s`**, **`--save`**:<br>
+  Save a page to the wayback machine (no download).
 
 #### Optional query parameters
 
@@ -219,7 +229,7 @@ Parameters will change the download behavior for snapshots.
   Set verbosity level. Available levels:
   - `low` (or `quiet`, `minimal`, `min`): Essential output only (same as no flag)
   - `default` (or `normal`, `verbose`): Standard verbose output (default when flag is set)
-  - `high` (or `debug`, `detailed`, `max`): Detailed verbose output
+  - `high` (or `detailed`, `max`): Detailed verbose output
   
   Examples: `--verbose`, `--verbose default`, `--verbose high`, `-v high`
 

{pywaybackup-4.1.5 → pywaybackup-4.2.0}/README.md

@@ -2,7 +2,7 @@
 
 [![PyPI](https://img.shields.io/pypi/v/pywaybackup)](https://pypi.org/project/pywaybackup/)
 [![PyPI - Downloads](https://img.shields.io/pypi/dm/pywaybackup)](https://pypi.org/project/pywaybackup/)
-![Python Version](https://img.shields.io/badge/Python-3.8-blue)
+![Python Version](https://img.shields.io/badge/Python-3.8%2B-blue)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 
 Downloading archived web pages from the [Wayback Machine](https://archive.org/web/).
@@ -31,6 +31,13 @@ This tool allows you to download content from the Wayback Machine (archive.org).
 2. Run the tool <br>
    `waybackup -h`
 
+### Standalone binary
+
+Prebuilt executables for Windows, Linux and macOS are attached to each [release](https://github.com/bitdruid/python-wayback-machine-downloader/releases). No Python required.
+
+- Run from a terminal with arguments like the pip version: `waybackup -h`
+- Or start it without arguments (e.g. double-click on Windows) to enter **interactive mode** — the tool will prompt you for URL, mode and optional settings.
+
 ### Manual
 
 1. Clone the repository <br>
@@ -118,8 +125,9 @@ output:
 
 ## cli
 
-- `-h`, `--help`: Show the help message and exit.
-- `-v`, `--version`: Show information about the tool and exit.
+- `-h`, `--help`: Show the help message and exit. Version info is shown in the help header.
+
+> **Interactive mode:** running `waybackup` without any arguments in a terminal starts a guided prompt for URL, mode and optional settings. Without a terminal (scripts/cron), the help is printed instead.
 
 #### Required
 
@@ -134,6 +142,8 @@ output:
   Last Version. Gives one folder containing the last version of each file of specified `--range`.
 - **`-f`**, **`--first`**:<br>
   First Version. Gives one folder containing the first version of each file of specified `--range`.
+- **`-s`**, **`--save`**:<br>
+  Save a page to the wayback machine (no download).
 
 #### Optional query parameters
 
@@ -182,7 +192,7 @@ Parameters will change the download behavior for snapshots.
   Set verbosity level. Available levels:
   - `low` (or `quiet`, `minimal`, `min`): Essential output only (same as no flag)
   - `default` (or `normal`, `verbose`): Standard verbose output (default when flag is set)
-  - `high` (or `debug`, `detailed`, `max`): Detailed verbose output
+  - `high` (or `detailed`, `max`): Detailed verbose output
   
   Examples: `--verbose`, `--verbose default`, `--verbose high`, `-v high`
 

{pywaybackup-4.1.5 → pywaybackup-4.2.0}/pyproject.toml

@@ -10,18 +10,18 @@ packages = ["pywaybackup"]
 [project]
 dependencies = [
     "ruff",
-    "SQLAlchemy==2.0.43",
+    "SQLAlchemy==2.0.51",
     "requests==2.32.3",
     "tqdm==4.67.1",
     "python-magic-standalone==0.4.28",
 ]
 name = "pywaybackup"
-version = "4.1.5"
+version = "4.2.0"
 description = "Query and download archive.org as simple as possible."
 authors = [{ name = "bitdruid", email = "bitdruid@outlook.com" }]
 license = { file = "LICENSE" }
 readme = "README.md"
-requires-python = ">=3.8,<3.14"
+requires-python = ">=3.8"
 
 
 [project.scripts]

{pywaybackup-4.1.5 → pywaybackup-4.2.0}/pywaybackup/PyWayBackup.py

@@ -364,6 +364,7 @@ class PyWayBackup:
         resources after the backup is complete.
 
         """
+        collection = None
         try:
             self._startup()
 
@@ -385,6 +386,12 @@ class PyWayBackup:
             self._keep = True
             ex.exception(message="", e=e)
         finally:
+            # if a collection was created during the workflow, close its DB session cleanly
+            try:
+                if collection:
+                    collection.close()
+            except Exception:
+                pass
             self._shutdown()
 
     def paths(self, rel: bool = False) -> dict:
@@ -503,6 +510,7 @@ class PyWayBackup:
         collection = SnapshotCollection()
         collection.close()
         self._csvfile.store_result()
+        db.close_engine()
         self._f_keep()
         vb.fini()
         signal.signal(signal.SIGINT, signal.SIG_IGN)

{pywaybackup-4.1.5 → pywaybackup-4.2.0}/pywaybackup/Snapshot.py

@@ -1,8 +1,9 @@
 import os
 import threading
 
-from pywaybackup.db import Database, select, update, waybackup_snapshots
+from pywaybackup.db import Database, select, update, waybackup_snapshots, and_
 from pywaybackup.helper import url_split
+from pywaybackup.Verbosity import Verbosity as vb
 
 
 class Snapshot:
@@ -70,20 +71,58 @@ class Snapshot:
             return False
 
         def __get_row():
-            with self._db.session.begin():
-                row = self._db.session.execute(
-                    select(waybackup_snapshots)
-                    .where(waybackup_snapshots.response.is_(None))
-                    .order_by(waybackup_snapshots.scid)
-                    .limit(1)
-                    .with_for_update(skip_locked=True)
-                ).scalar_one_or_none()
-
-                if row is None:
-                    return None
-
-                row.response = "LOCK"
+            # Atomic claim: find next unprocessed scid, set response='LOCK' only if still unprocessed,
+            # then fetch that row. This avoids relying on FOR UPDATE or explicit nested transactions
+            # which can trigger "A transaction is already begun on this Session" errors.
+
+            session = self._db.session
+
+            # get next available SnapshotId
+            vb.write(verbose="high", content="[Snapshot.fetch] selecting next scid")
+            scid = session.execute(
+                select(waybackup_snapshots.scid)
+                .where(waybackup_snapshots.response.is_(None))
+                .order_by(waybackup_snapshots.scid)
+                .limit(1)
+            ).scalar_one_or_none()
+
+            if scid is None:
+                vb.write(verbose="high", content="[Snapshot.fetch] no unprocessed scid found")
+                return None
+
+            # try to atomically claim the row by updating only if still unclaimed
+            result = session.execute(
+                update(waybackup_snapshots)
+                .where(and_(waybackup_snapshots.scid == scid, waybackup_snapshots.response.is_(None)))
+                .values(response="LOCK")
+            )
+
+            # if another worker claimed it first, rowcount will be 0 — retry to get next available row
+            vb.write(
+                verbose="high", content=f"[Snapshot.fetch] attempted to claim scid={scid}, rowcount={result.rowcount}"
+            )
+            if result.rowcount == 0:
+                # TOCTOU: __get_row(): another worker claimed this row between our SELECT and UPDATE.
+                # Retry instead of returning None to avoid premature worker termination.
+                try:
+                    session.commit()
+                except Exception:
+                    pass
+                vb.write(verbose="high", content=f"[Snapshot.fetch] scid={scid} already claimed, retrying")
+                return __get_row()
 
+            # The row has been claimed by the worker and can now be fetched.
+            row = session.execute(
+                select(waybackup_snapshots).where(waybackup_snapshots.scid == scid)
+            ).scalar_one_or_none()
+            try:
+                session.commit()
+            except Exception:
+                try:
+                    session.rollback()
+                except Exception:
+                    pass
+            vb.write(verbose="high", content=f"[Snapshot.fetch] claimed scid={scid} and fetched row")
             return row
 
         if __on_sqlite():
@@ -101,10 +140,22 @@ class Snapshot:
             value: New value to set for the column.
         """
         column = getattr(waybackup_snapshots, column)
-        self._db.session.execute(
-            update(waybackup_snapshots).where(waybackup_snapshots.scid == self.scid).values({column: value})
-        )
-        self._db.session.commit()
+        try:
+            vb.write(verbose="high", content=f"[Snapshot.modify] updating scid={self.scid} column={column.key}")
+            self._db.session.execute(
+                update(waybackup_snapshots).where(waybackup_snapshots.scid == self.scid).values({column: value})
+            )
+            self._db.session.commit()
+            vb.write(verbose="high", content=f"[Snapshot.modify] update committed scid={self.scid} column={column.key}")
+        except Exception as e:
+            vb.write(
+                verbose="high", content=f"[Snapshot.modify] update failed scid={self.scid} error={e}; rolling back"
+            )
+            try:
+                self._db.session.rollback()
+            except Exception:
+                pass
+            raise
 
     def create_output(self):
         """

{pywaybackup-4.1.5 → pywaybackup-4.2.0}/pywaybackup/SnapshotCollection.py

@@ -1,6 +1,6 @@
 import json
 
-from pywaybackup.db import Database, Index, and_, delete, func, or_, select, tuple_, update, waybackup_snapshots
+from pywaybackup.db import Database, and_, delete, func, or_, select, text, tuple_, update, waybackup_snapshots
 from pywaybackup.files import CDXfile, CSVfile
 from pywaybackup.Verbosity import Progressbar
 from pywaybackup.Verbosity import Verbosity as vb
@@ -163,73 +163,89 @@ class SnapshotCollection:
 
         vb.write(verbose=None, content="\nInserting CDX data into database...")
 
-        progressbar = Progressbar(
-            unit=" lines",
-            total=self._cdx_total,
-            desc="process cdx".ljust(15),
-            ascii="░▒█",
-            bar_format="{l_bar}{bar:50}{r_bar}{bar:-10b}",
-        )
-        line_batchsize = 2500
-        line_batch = []
-        total_inserted = 0
-        first_line = True
-
-        with self.cdxfile as f:
-            for line in f:
-                if first_line:
-                    first_line = False
-                    continue
-                line = line.strip()
-                if line.endswith("]]"):
-                    line = line.rsplit("]", 1)[0]
-                if line.endswith(","):
-                    line = line.rsplit(",", 1)[0]
-
-                try:
-                    line_batch.append(__parse_line(line))
-                except json.decoder.JSONDecodeError:
-                    self._snapshot_faulty += 1
-                    continue
-
-                if len(line_batch) >= line_batchsize:
+        try:
+            vb.write(verbose=True, content="[SnapshotCollection._insert_cdx] starting insert_cdx operation")
+            progressbar = Progressbar(
+                unit=" lines",
+                total=self._cdx_total,
+                desc="process cdx".ljust(15),
+                ascii="░▒█",
+                bar_format="{l_bar}{bar:50}{r_bar}{bar:-10b}",
+            )
+            line_batchsize = 2500
+            line_batch = []
+            total_inserted = 0
+            first_line = True
+
+            with self.cdxfile as f:
+                for line in f:
+                    if first_line:
+                        first_line = False
+                        continue
+                    line = line.strip()
+                    if line.endswith("]]"):
+                        line = line.rsplit("]", 1)[0]
+                    if line.endswith(","):
+                        line = line.rsplit(",", 1)[0]
+
+                    try:
+                        line_batch.append(__parse_line(line))
+                    except json.decoder.JSONDecodeError:
+                        self._snapshot_faulty += 1
+                        continue
+
+                    if len(line_batch) >= line_batchsize:
+                        total_inserted += _insert_batch_safe(line_batch=line_batch)
+                        line_batch = []
+                        progressbar.update(line_batchsize)
+
+                if line_batch:
                     total_inserted += _insert_batch_safe(line_batch=line_batch)
-                    line_batch = []
-                    progressbar.update(line_batchsize)
-
-            if line_batch:
-                total_inserted += _insert_batch_safe(line_batch=line_batch)
-                progressbar.update(len(line_batch))
-
-        self.db.session.commit()
+                    progressbar.update(len(line_batch))
+
+            self.db.session.commit()
+            vb.write(verbose=True, content="[SnapshotCollection._insert_cdx] insert_cdx commit successful")
+        except Exception as e:
+            vb.write(verbose=True, content=f"[SnapshotCollection._insert_cdx] exception: {e}; rolling back")
+            try:
+                self.db.session.rollback()
+                vb.write(verbose=True, content="[SnapshotCollection._insert_cdx] rollback successful")
+            except Exception:
+                vb.write(verbose=True, content="[SnapshotCollection._insert_cdx] rollback failed")
+            raise
 
     def _index_snapshots(self):
         """
         Create indexes for the snapshot table.
+
+        Raw DDL instead of sqlalchemy Index objects: Index(...) attaches to the
+        module-global table metadata, which accumulates duplicates when the
+        package is reused in-process (library usage) and breaks create_all().
         """
         # index for filtering last snapshots
         if self._mode_last:
-            idx1 = Index(
-                "idx_waybackup_snapshots_url_origin_timestamp_desc",
-                waybackup_snapshots.url_origin,
-                waybackup_snapshots.timestamp.desc(),
+            self.db.session.execute(
+                text(
+                    "CREATE INDEX IF NOT EXISTS idx_waybackup_snapshots_url_origin_timestamp_desc "
+                    "ON waybackup_snapshots (url_origin, timestamp DESC)"
+                )
             )
-            idx1.create(self.db.session.bind, checkfirst=True)
         # index for filtering first snapshots
         if self._mode_first:
-            idx2 = Index(
-                "idx_waybackup_snapshots_url_origin_timestamp_asc",
-                waybackup_snapshots.url_origin,
-                waybackup_snapshots.timestamp.asc(),
+            self.db.session.execute(
+                text(
+                    "CREATE INDEX IF NOT EXISTS idx_waybackup_snapshots_url_origin_timestamp_asc "
+                    "ON waybackup_snapshots (url_origin, timestamp ASC)"
+                )
             )
-            idx2.create(self.db.session.bind, checkfirst=True)
         # index for skippable snapshots
-        idx3 = Index(
-            "idx_waybackup_snapshots_timestamp_url_origin_response",
-            waybackup_snapshots.timestamp,
-            waybackup_snapshots.url_origin,
+        self.db.session.execute(
+            text(
+                "CREATE INDEX IF NOT EXISTS idx_waybackup_snapshots_timestamp_url_origin_response "
+                "ON waybackup_snapshots (timestamp, url_origin)"
+            )
         )
-        idx3.create(self.db.session.bind, checkfirst=True)
+        self.db.session.commit()
 
     def _filter_snapshots(self):
         """
@@ -297,29 +313,42 @@ class SnapshotCollection:
         """
 
         # ? for now per row / no bulk for compatibility
-        with self.csvfile as f:
-            total_skipped = 0
-            for row in f:
-                self.db.session.execute(
-                    update(waybackup_snapshots)
-                    .where(
-                        and_(
-                            waybackup_snapshots.timestamp == row["timestamp"],
-                            waybackup_snapshots.url_origin == row["url_origin"],
+        try:
+            vb.write(verbose=True, content="[SnapshotCollection._skip_set] applying CSV skips to DB")
+            with self.csvfile as f:
+                total_skipped = 0
+                for row in f:
+                    self.db.session.execute(
+                        update(waybackup_snapshots)
+                        .where(
+                            and_(
+                                waybackup_snapshots.timestamp == row["timestamp"],
+                                waybackup_snapshots.url_origin == row["url_origin"],
+                            )
+                        )
+                        .values(
+                            url_archive=row["url_archive"],
+                            redirect_url=row["redirect_url"],
+                            redirect_timestamp=row["redirect_timestamp"],
+                            response=row["response"],
+                            file=row["file"],
                         )
                     )
-                    .values(
-                        url_archive=row["url_archive"],
-                        redirect_url=row["redirect_url"],
-                        redirect_timestamp=row["redirect_timestamp"],
-                        response=row["response"],
-                        file=row["file"],
-                    )
-                )
-                total_skipped += 1
+                    total_skipped += 1
 
-        self.db.session.commit()
-        self._filter_skip = total_skipped
+            self.db.session.commit()
+            self._filter_skip = total_skipped
+            vb.write(
+                verbose=True, content=f"[SnapshotCollection._skip_set] commit successful, total_skipped={total_skipped}"
+            )
+        except Exception as e:
+            vb.write(verbose=True, content=f"[SnapshotCollection._skip_set] exception: {e}; rolling back")
+            try:
+                self.db.session.rollback()
+                vb.write(verbose=True, content="[SnapshotCollection._skip_set] rollback successful")
+            except Exception:
+                vb.write(verbose=True, content="[SnapshotCollection._skip_set] rollback failed")
+            raise
 
     def count_total(self) -> int:
         return self.db.session.query(waybackup_snapshots.scid).count()

{pywaybackup-4.1.5 → pywaybackup-4.2.0}/pywaybackup/Worker.py

@@ -21,6 +21,27 @@ class Worker:
         self.db = Database()
         self.connection = http.client.HTTPSConnection("web.archive.org")
 
+    def close(self):
+        """
+        Try to close the database and connection.
+        """
+        try:
+            if hasattr(self, "db") and self.db:
+                try:
+                    vb.write(verbose="high", content=f"[Worker.close] closing DB for worker {self.id}")
+                    self.db.close()
+                    vb.write(verbose="high", content=f"[Worker.close] DB closed for worker {self.id}")
+                except Exception:
+                    pass
+        finally:
+            try:
+                if hasattr(self, "connection") and self.connection:
+                    vb.write(verbose="high", content=f"[Worker.close] closing connection for worker {self.id}")
+                    self.connection.close()
+                    vb.write(verbose="high", content=f"[Worker.close] connection closed for worker {self.id}")
+            except Exception:
+                pass
+
     def assign_snapshot(self, total_amount: int):
         self.snapshot = Snapshot(self.db, output=self.output, mode=self.mode)
         self.total_amount = total_amount
@@ -84,7 +105,11 @@ class Message(Worker):
             content = content + " - " if content else ""
             self.message = {
                 "verbose": False,
-                "content": f"{self.worker.snapshot.counter}/{self.worker.total_amount} - W:{self.worker.id} - {result}{content}{self.worker.snapshot.timestamp} - {self.worker.snapshot.url_origin}",
+                "content": (
+                    f"{self.worker.snapshot.counter}/{self.worker.total_amount}"
+                    f" - W:{self.worker.id} - {result}{content}"
+                    f"{self.worker.snapshot.timestamp} - {self.worker.snapshot.url_origin}"
+                ),
             }
             self.buffer.append(self.message)
 

{pywaybackup-4.1.5 → pywaybackup-4.2.0}/pywaybackup/archive_download.py

@@ -267,6 +267,8 @@ class DownloadArchive:
 
         except Exception as e:
             ex.exception(f"\nWorker: {worker.id} - Exception", e)
+        finally:
+            worker.close()  
 
     def _download(self, worker: Worker):
         """

pywaybackup-4.2.0/pywaybackup/arg_parser.py

@@ -0,0 +1,55 @@
+import argparse
+import sys
+from argparse import RawTextHelpFormatter
+from importlib.metadata import version
+
+from pywaybackup.arg_specs import ARG_GROUPS, ARG_SPECS, EXCLUSIVE_GROUPS
+
+
+class Arguments:
+    def __init__(self):
+        parser = argparse.ArgumentParser(
+            description=f"<<< python-wayback-machine-downloader v{version('pywaybackup')} >>>\nby @bitdruid -> https://github.com/bitdruid",
+            formatter_class=RawTextHelpFormatter,
+        )
+
+        groups = {name: parser.add_argument_group(name) for name in ARG_GROUPS}
+
+        exclusive = {
+            ex_name: groups[ex_meta["parent_group"]].add_mutually_exclusive_group(required=ex_meta["required"])
+            for ex_name, ex_meta in EXCLUSIVE_GROUPS.items()
+        }
+
+        for spec in ARG_SPECS:
+            target = exclusive[spec.exclusive_group] if spec.exclusive_group else groups[spec.group]
+            target.add_argument(*spec.flags, **_argparse_kwargs(spec))
+
+        args = parser.parse_args(args=None if sys.argv[1:] else ["--help"])  # if no arguments are given, print help
+
+        args.silent = False
+        args.debug = True
+
+        self.args = args
+
+    def get_args(self) -> dict:
+        """Returns the parsed arguments as a dictionary."""
+        return vars(self.args)
+
+
+def _argparse_kwargs(spec) -> dict:
+    """Translate an ArgSpec into kwargs for argparse.add_argument()."""
+    kwargs = {"help": spec.help}
+    if spec.action == "store_true":
+        kwargs["action"] = "store_true"
+        kwargs["default"] = bool(spec.default)
+    elif spec.action == "optional_value":
+        kwargs["type"] = spec.type
+        kwargs["nargs"] = "?"
+        kwargs["const"] = spec.const
+        kwargs["metavar"] = spec.metavar
+        kwargs["default"] = spec.default
+    else:
+        kwargs["type"] = spec.type
+        kwargs["metavar"] = spec.metavar
+        kwargs["default"] = spec.default
+    return kwargs