pywaybackup 3.1.0__tar.gz → 3.2.1__tar.gz

{pywaybackup-3.1.0/pywaybackup.egg-info → pywaybackup-3.2.1}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.2
+Metadata-Version: 2.4
 Name: pywaybackup
-Version: 3.1.0
+Version: 3.2.1
 Summary: Query and download archive.org as simple as possible.
 Author-email: bitdruid <bitdruid@outlook.com>
 License: MIT License
@@ -29,18 +29,19 @@ Project-URL: homepage, https://github.com/bitdruid/python-wayback-machine-downlo
 Requires-Python: >=3.8
 Description-Content-Type: text/markdown
 License-File: LICENSE
-Requires-Dist: pysqlite3-binary==0.5.4
-Requires-Dist: requests==2.31.0
-Requires-Dist: tqdm==4.66.2
+Requires-Dist: pysqlite3-binary==0.5.4; sys_platform == "linux"
+Requires-Dist: pysqlite-binary; sys_platform == "win32"
+Requires-Dist: requests==2.32.3
+Requires-Dist: tqdm==4.67.1
 Requires-Dist: python-magic==0.4.27; sys_platform == "linux"
 Requires-Dist: python-magic-bin==0.4.14; sys_platform == "win32"
+Dynamic: license-file
 
 # python wayback machine downloader
 
 [![PyPI](https://img.shields.io/pypi/v/pywaybackup)](https://pypi.org/project/pywaybackup/)
 [![PyPI - Downloads](https://img.shields.io/pypi/dm/pywaybackup)](https://pypi.org/project/pywaybackup/)
 ![Python Version](https://img.shields.io/badge/Python-3.8-blue)
-<!-- ![Python_Sqlite3 Version](https://img.shields.io/badge/Python_Sqlite3-3.35-blue) -->
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 
 Downloading archived web pages from the [Wayback Machine](https://archive.org/web/).
@@ -66,11 +67,15 @@ This tool allows you to download content from the Wayback Machine (archive.org).
    ```pip install .```
    - in a virtual env or use `--break-system-package`
 
-## Usage infos - important notes
+## notes / issues / hints
 
-- Linux recommended: On Windows machines, the path length is limited. This can only be overcome by editing the registry. Files that exceed the path length will not be downloaded.
-- If you query an explicit file (e.g. a query-string `?query=this` or `login.html`), the `--explicit`-argument is recommended as a wildcard query may lead to an empty result.
+- Linux recommended: On Windows machines, the path length is limited. Files that exceed the path length will not be downloaded.
 - The tool uses a sqlite database to handle snapshots. The database will only persist while the download is running.
+- If you query an explicit file (e.g. a query-string `?query=this` or `login.html`), the `--explicit`-argument is recommended as a wildcard query may lead to an empty result.
+- Downloading directly into a network share is not recommended. The sqlite locking mechanism may cause issues. If you need to download into a network share, set the `--metadata` argument to a local path.
+
+<br>
+<br>
 
 ## Arguments
 
@@ -92,7 +97,7 @@ This tool allows you to download content from the Wayback Machine (archive.org).
 - **`-s`**, **`--save`**:<br>
   Save a page to the Wayback Machine. (beta)
 
-### Optional query parameters
+#### Optional query parameters
 
 - **`-e`**, **`--explicit`**:<br>
   Only download the explicit given URL. No wildcard subdomains or paths. Use e.g. to get root-only snapshots. This is recommended for explicit files like `login.html` or `?query=this`.
@@ -113,11 +118,16 @@ Limits the amount of snapshots to query from the CDX server. If an existing CDX
    - **`--end`**:<br>
      Timestamp to end searching.
 
-### Behavior manipulation
+### Optional
+
+#### Behavior Manipulation
 
 - **`-o`**, **`--output`**:<br>
 Defaults to `waybackup_snapshots` in the current directory. The folder where downloaded files will be saved.
 
+- **`-m`**, **`--metadata`**<br>
+Change the folder where metadata will be saved (`cdx`/`db`/`csv`/`log`). Especially if you are downloading into a network share, you SHOULD set this to a local path because sqlite locking mechanism may cause issues with network shares.
+
 <!-- - **`--verbosity`** `<level>`:<br>
 Sets verbosity level. Options are `info`and `trace`. Default is `info`. -->
 
@@ -132,17 +142,31 @@ Sets the number of simultaneous download workers. Default is 1, safe range is ab
 
 - **`--no-redirect`**:<br>
 Disables following redirects of snapshots. Useful for preventing timestamp-folder mismatches caused by Archive.org redirects.
-  
+
 - **`--retry`** `<attempts>`:<br>
 Specifies number of retry attempts for failed downloads.
 
 - **`--delay`** `<seconds>`:<br>
 Specifies delay between download requests in seconds. Default is no delay (0).
 
+- **`--verbose`**:<br>
+Increase output verbosity.
+  - verbose:
+  ```
+  -----> Worker: 2 - Attempt: [1/1] Snapshot ID: [23/81]
+  SUCCESS   -> 200 OK
+            -> URL:  https://web.archive.org/web/20240225193302id_/https://example.com/assets/css/custom-styles.css
+            -> FILE: /home/manjaro/Stuff/python-wayback-machine-downloader/waybackup_snapshots/example.com/20240225193302id_/assets/css/custom-styles.css
+  ```
+  - non-verbose:
+  ```
+  55/81 - W:2 - SUCCESS - 20240225193302 - https://example.com/assets/css/custom-styles.css
+  ```
+
 <!-- - **`--convert-links`**:<br>
 If set, all links in the downloaded files will be converted to local links. This is useful for offline browsing. The links are converted to the local path structure. Show output with `--verbosity trace`. -->
 
-### Special:
+#### Job Handling:
 
 - **`--reset`**:  
   If set, the job will be reset, and any existing `cdx`, `db`, `csv` files will be **deleted**. This allows you to start the job from scratch without considering previously downloaded data.
@@ -150,47 +174,56 @@ If set, all links in the downloaded files will be converted to local links. This
 - **`--keep`**:  
   If set, all files will be kept after the job is finished. This includes the `cdx` and `db` file. Without this argument, they will be deleted if the job finished successfully.
 
-# Usage 
+<br>
+<br>
+
+## Usage
 
 ### Handling Interrupted Jobs
-When a job is interrupted (by any reason), `pywaybackup` is designed to resume the job from where it left off. It automatically detects existing job data (based on the URL and <u>**optional query parameters**</u> - including output directory) and resumes the process without requiring manual intervention. Here's how the tool handles different scenarios:
-
-- **Default Behavior:** 
-  - On restarting the same job (same URL, <u>**optional query parameters**</u>, and output directory), the tool will:
-    - Reuse the existing `.cdx` and `.db` files.
-    - Resume downloading snapshots from the last successful point.
-    - Skip previously downloaded files to save time and resources.
-
-- **Manual Reset with `--reset`:** 
-  - This command deletes any existing `.cdx` and `.db` files associated with the job and starts the process from scratch.
-  - Useful if:
-    - The previous data is corrupted.
-    - You want to re-query the snapshots without considering previously downloaded data.
-
-- **Preserving Job Data with `--keep`:** 
-  - Normally, `.cdx` and `.db` files are deleted after the job finishes successfully.
-  - Use `--keep` to retain these files for future use (e.g., re-analysis or extending the query later).
-
-> **Note1:** The resumption process only works if the output directory remains the same as the one used during the initial job.
-> 
-> **Note2:** `--reset` will NOT delete the already downloaded files for now. You have to remove them 'by hand'.
-  
-### Example
-
-1. Start downloading all available snapshots:<br>`waybackup -u https://example.com -a`
-2. Interrupt the process `CTRL+C`<br>
-3. The tool will detect the existing job data and resume downloading from the last completed point:<br>`waybackup -u https://example.com -a`
-> **Important:** `waybackup -u https://example.com -c` -> The tool will NOT resume because a necessary identifier-changed
-4. This deletes any existing .cdx and .db files associated with the job and starts the process from scratch:<br>`waybackup -u https://example.com -a --reset`
-5. This ensures all job-related files are kept for future use, such as re-analysis or extending the query later:<br>`waybackup -u https://example.com -a --keep`
-
-## Output path structure
+
+`pywaybackup` resumes interrupted jobs. The tool automatically continues from where it left off.
+
+- Detects existing `.cdx` and `.db` files in an `output dir` to resume downloading from the last successful point.
+- Compares `URL`, `mode`, and `optional query parameters` to ensure automatic resumption.
+- Skips previously downloaded files to save time.
+> **Note:** Changing URL, mode selection, query parameters or output prevents automatic resumption.
+
+#### Resetting a Job (`--reset`)
+- Deletes `.cdx` and `.db` files and restarts the process from scratch.
+- Does **not** remove already downloaded files.
+- `waybackup -u https://example.com -a --reset`
+
+#### Keeping Job Data (`--keep`)
+- Normally, `.cdx` and `.db` files are deleted after a successful job.
+- `--keep` preserves them for future re-analysis or extending the query.
+- `waybackup -u https://example.com -a --keep`
+
+<br>
+<br>
+
+## Examples
+
+1. Download a specific single snapshot of all available files (starting from root):<br>
+`waybackup -u https://example.com -a --start 20210101000000 --end 20210101000000`
+2. Download a specific single snapshot of all available files (starting from a subdirectory):<br>
+`waybackup -u https://example.com/subdir1/subdir2/assets/ -a --start 20210101000000 --end 20210101000000`
+3. Download a specific single snapshot of the exact given URL (no subdirs):<br>
+`waybackup -u https://example.com -a --start 20210101000000 --end 20210101000000 --explicit`
+4. Download all snapshots of all available files in the given range:<br>
+`waybackup -u https://example.com -a --start 20210101000000 --end 20231122000000`
+
+<br>
+<br>
+
+## Output
+
+### Path Structure
 
 The output path is currently structured as follows by an example for the query:<br>
-`http://example.com/subdir1/subdir2/assets/`:
+`http://example.com/subdir1/subdir2/assets/`
 <br><br>
 For the first and last version (`-f` or `-l`):
-- The requested path will only include all files/folders starting from your query-path.
+- Will only include all files/folders starting from your query-path.
 ```
 your/path/waybackup_snapshots/
 └── the_root_of_your_query/ (example.com/)
@@ -202,7 +235,7 @@ your/path/waybackup_snapshots/
                 ...
 ```
 For all versions (`-a`):
-- Will currently create a folder named as the root of your query. Inside this folder, you will find all timestamps and per timestamp the path you requested.
+- Will create a folder named as the root of your query. Inside this folder, you will find all timestamps and per timestamp the path you requested.
 ```
 your/path/waybackup_snapshots/
 └── the_root_of_your_query/ (example.com/)
@@ -221,7 +254,7 @@ your/path/waybackup_snapshots/
     ...
 ```
 
-## CSV Output
+### CSV
 
 Each snapshot is stored with the following keys/values. These are either stored in a sqlite database while the download is running or saved into a CSV file after the download is finished.
 
@@ -247,9 +280,8 @@ For download queries:
 
 Exceptions will be written into `waybackup_error.log` (each run overwrites the file).
 
-### Known ToDos
-
-- [ ] currently there is no logic to handle if both a http and https version of a page is available
+<br>
+<br>
 
 ## Contributing
 

{pywaybackup-3.1.0 → pywaybackup-3.2.1}/README.md

@@ -3,7 +3,6 @@
 [![PyPI](https://img.shields.io/pypi/v/pywaybackup)](https://pypi.org/project/pywaybackup/)
 [![PyPI - Downloads](https://img.shields.io/pypi/dm/pywaybackup)](https://pypi.org/project/pywaybackup/)
 ![Python Version](https://img.shields.io/badge/Python-3.8-blue)
-<!-- ![Python_Sqlite3 Version](https://img.shields.io/badge/Python_Sqlite3-3.35-blue) -->
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 
 Downloading archived web pages from the [Wayback Machine](https://archive.org/web/).
@@ -29,11 +28,15 @@ This tool allows you to download content from the Wayback Machine (archive.org).
    ```pip install .```
    - in a virtual env or use `--break-system-package`
 
-## Usage infos - important notes
+## notes / issues / hints
 
-- Linux recommended: On Windows machines, the path length is limited. This can only be overcome by editing the registry. Files that exceed the path length will not be downloaded.
-- If you query an explicit file (e.g. a query-string `?query=this` or `login.html`), the `--explicit`-argument is recommended as a wildcard query may lead to an empty result.
+- Linux recommended: On Windows machines, the path length is limited. Files that exceed the path length will not be downloaded.
 - The tool uses a sqlite database to handle snapshots. The database will only persist while the download is running.
+- If you query an explicit file (e.g. a query-string `?query=this` or `login.html`), the `--explicit`-argument is recommended as a wildcard query may lead to an empty result.
+- Downloading directly into a network share is not recommended. The sqlite locking mechanism may cause issues. If you need to download into a network share, set the `--metadata` argument to a local path.
+
+<br>
+<br>
 
 ## Arguments
 
@@ -55,7 +58,7 @@ This tool allows you to download content from the Wayback Machine (archive.org).
 - **`-s`**, **`--save`**:<br>
   Save a page to the Wayback Machine. (beta)
 
-### Optional query parameters
+#### Optional query parameters
 
 - **`-e`**, **`--explicit`**:<br>
   Only download the explicit given URL. No wildcard subdomains or paths. Use e.g. to get root-only snapshots. This is recommended for explicit files like `login.html` or `?query=this`.
@@ -76,11 +79,16 @@ Limits the amount of snapshots to query from the CDX server. If an existing CDX
    - **`--end`**:<br>
      Timestamp to end searching.
 
-### Behavior manipulation
+### Optional
+
+#### Behavior Manipulation
 
 - **`-o`**, **`--output`**:<br>
 Defaults to `waybackup_snapshots` in the current directory. The folder where downloaded files will be saved.
 
+- **`-m`**, **`--metadata`**<br>
+Change the folder where metadata will be saved (`cdx`/`db`/`csv`/`log`). Especially if you are downloading into a network share, you SHOULD set this to a local path because sqlite locking mechanism may cause issues with network shares.
+
 <!-- - **`--verbosity`** `<level>`:<br>
 Sets verbosity level. Options are `info`and `trace`. Default is `info`. -->
 
@@ -95,17 +103,31 @@ Sets the number of simultaneous download workers. Default is 1, safe range is ab
 
 - **`--no-redirect`**:<br>
 Disables following redirects of snapshots. Useful for preventing timestamp-folder mismatches caused by Archive.org redirects.
-  
+
 - **`--retry`** `<attempts>`:<br>
 Specifies number of retry attempts for failed downloads.
 
 - **`--delay`** `<seconds>`:<br>
 Specifies delay between download requests in seconds. Default is no delay (0).
 
+- **`--verbose`**:<br>
+Increase output verbosity.
+  - verbose:
+  ```
+  -----> Worker: 2 - Attempt: [1/1] Snapshot ID: [23/81]
+  SUCCESS   -> 200 OK
+            -> URL:  https://web.archive.org/web/20240225193302id_/https://example.com/assets/css/custom-styles.css
+            -> FILE: /home/manjaro/Stuff/python-wayback-machine-downloader/waybackup_snapshots/example.com/20240225193302id_/assets/css/custom-styles.css
+  ```
+  - non-verbose:
+  ```
+  55/81 - W:2 - SUCCESS - 20240225193302 - https://example.com/assets/css/custom-styles.css
+  ```
+
 <!-- - **`--convert-links`**:<br>
 If set, all links in the downloaded files will be converted to local links. This is useful for offline browsing. The links are converted to the local path structure. Show output with `--verbosity trace`. -->
 
-### Special:
+#### Job Handling:
 
 - **`--reset`**:  
   If set, the job will be reset, and any existing `cdx`, `db`, `csv` files will be **deleted**. This allows you to start the job from scratch without considering previously downloaded data.
@@ -113,47 +135,56 @@ If set, all links in the downloaded files will be converted to local links. This
 - **`--keep`**:  
   If set, all files will be kept after the job is finished. This includes the `cdx` and `db` file. Without this argument, they will be deleted if the job finished successfully.
 
-# Usage 
+<br>
+<br>
+
+## Usage
 
 ### Handling Interrupted Jobs
-When a job is interrupted (by any reason), `pywaybackup` is designed to resume the job from where it left off. It automatically detects existing job data (based on the URL and <u>**optional query parameters**</u> - including output directory) and resumes the process without requiring manual intervention. Here's how the tool handles different scenarios:
-
-- **Default Behavior:** 
-  - On restarting the same job (same URL, <u>**optional query parameters**</u>, and output directory), the tool will:
-    - Reuse the existing `.cdx` and `.db` files.
-    - Resume downloading snapshots from the last successful point.
-    - Skip previously downloaded files to save time and resources.
-
-- **Manual Reset with `--reset`:** 
-  - This command deletes any existing `.cdx` and `.db` files associated with the job and starts the process from scratch.
-  - Useful if:
-    - The previous data is corrupted.
-    - You want to re-query the snapshots without considering previously downloaded data.
-
-- **Preserving Job Data with `--keep`:** 
-  - Normally, `.cdx` and `.db` files are deleted after the job finishes successfully.
-  - Use `--keep` to retain these files for future use (e.g., re-analysis or extending the query later).
-
-> **Note1:** The resumption process only works if the output directory remains the same as the one used during the initial job.
-> 
-> **Note2:** `--reset` will NOT delete the already downloaded files for now. You have to remove them 'by hand'.
-  
-### Example
-
-1. Start downloading all available snapshots:<br>`waybackup -u https://example.com -a`
-2. Interrupt the process `CTRL+C`<br>
-3. The tool will detect the existing job data and resume downloading from the last completed point:<br>`waybackup -u https://example.com -a`
-> **Important:** `waybackup -u https://example.com -c` -> The tool will NOT resume because a necessary identifier-changed
-4. This deletes any existing .cdx and .db files associated with the job and starts the process from scratch:<br>`waybackup -u https://example.com -a --reset`
-5. This ensures all job-related files are kept for future use, such as re-analysis or extending the query later:<br>`waybackup -u https://example.com -a --keep`
-
-## Output path structure
+
+`pywaybackup` resumes interrupted jobs. The tool automatically continues from where it left off.
+
+- Detects existing `.cdx` and `.db` files in an `output dir` to resume downloading from the last successful point.
+- Compares `URL`, `mode`, and `optional query parameters` to ensure automatic resumption.
+- Skips previously downloaded files to save time.
+> **Note:** Changing URL, mode selection, query parameters or output prevents automatic resumption.
+
+#### Resetting a Job (`--reset`)
+- Deletes `.cdx` and `.db` files and restarts the process from scratch.
+- Does **not** remove already downloaded files.
+- `waybackup -u https://example.com -a --reset`
+
+#### Keeping Job Data (`--keep`)
+- Normally, `.cdx` and `.db` files are deleted after a successful job.
+- `--keep` preserves them for future re-analysis or extending the query.
+- `waybackup -u https://example.com -a --keep`
+
+<br>
+<br>
+
+## Examples
+
+1. Download a specific single snapshot of all available files (starting from root):<br>
+`waybackup -u https://example.com -a --start 20210101000000 --end 20210101000000`
+2. Download a specific single snapshot of all available files (starting from a subdirectory):<br>
+`waybackup -u https://example.com/subdir1/subdir2/assets/ -a --start 20210101000000 --end 20210101000000`
+3. Download a specific single snapshot of the exact given URL (no subdirs):<br>
+`waybackup -u https://example.com -a --start 20210101000000 --end 20210101000000 --explicit`
+4. Download all snapshots of all available files in the given range:<br>
+`waybackup -u https://example.com -a --start 20210101000000 --end 20231122000000`
+
+<br>
+<br>
+
+## Output
+
+### Path Structure
 
 The output path is currently structured as follows by an example for the query:<br>
-`http://example.com/subdir1/subdir2/assets/`:
+`http://example.com/subdir1/subdir2/assets/`
 <br><br>
 For the first and last version (`-f` or `-l`):
-- The requested path will only include all files/folders starting from your query-path.
+- Will only include all files/folders starting from your query-path.
 ```
 your/path/waybackup_snapshots/
 └── the_root_of_your_query/ (example.com/)
@@ -165,7 +196,7 @@ your/path/waybackup_snapshots/
                 ...
 ```
 For all versions (`-a`):
-- Will currently create a folder named as the root of your query. Inside this folder, you will find all timestamps and per timestamp the path you requested.
+- Will create a folder named as the root of your query. Inside this folder, you will find all timestamps and per timestamp the path you requested.
 ```
 your/path/waybackup_snapshots/
 └── the_root_of_your_query/ (example.com/)
@@ -184,7 +215,7 @@ your/path/waybackup_snapshots/
     ...
 ```
 
-## CSV Output
+### CSV
 
 Each snapshot is stored with the following keys/values. These are either stored in a sqlite database while the download is running or saved into a CSV file after the download is finished.
 
@@ -210,11 +241,10 @@ For download queries:
 
 Exceptions will be written into `waybackup_error.log` (each run overwrites the file).
 
-### Known ToDos
-
-- [ ] currently there is no logic to handle if both a http and https version of a page is available
+<br>
+<br>
 
 ## Contributing
 
 I'm always happy for some feature requests to improve the usability of this tool.
-Feel free to give suggestions and report issues. Project is still far from being perfect.
+Feel free to give suggestions and report issues. Project is still far from being perfect.

{pywaybackup-3.1.0 → pywaybackup-3.2.1}/pyproject.toml

@@ -7,7 +7,7 @@ packages = ["pywaybackup"]
 
 [project]
 name = "pywaybackup"
-version = "3.1.0"
+version = "3.2.1"
 description = "Query and download archive.org as simple as possible."
 authors = [
     { name = "bitdruid", email = "bitdruid@outlook.com" }
@@ -16,9 +16,10 @@ license = { file = "LICENSE" }
 readme = "README.md"
 requires-python = ">=3.8"
 dependencies = [
-    "pysqlite3-binary==0.5.4",
-    "requests==2.31.0",
-    "tqdm==4.66.2",
+    "pysqlite3-binary==0.5.4; sys_platform == 'linux'",
+    "pysqlite-binary; sys_platform == 'win32'",
+    "requests==2.32.3",
+    "tqdm==4.67.1",
     "python-magic==0.4.27; sys_platform == 'linux'",
     "python-magic-bin==0.4.14; sys_platform == 'win32'",
 ]

{pywaybackup-3.1.0 → pywaybackup-3.2.1}/pywaybackup/Arguments.py

@@ -6,15 +6,14 @@ import argparse
 from importlib.metadata import version
 
 from pywaybackup.helper import url_split, sanitize_filename
-from pywaybackup.Exception import Exception as ex
 
 class Arguments:
-    
+
     def __init__(self):
-        
+
         parser = argparse.ArgumentParser(description='Download from wayback machine (archive.org)')
         parser.add_argument('-v', '--version', action='version', version='%(prog)s ' + version("pywaybackup") + ' by @bitdruid -> https://github.com/bitdruid')
-        
+
         required = parser.add_argument_group('required (one exclusive)')
         required.add_argument('-u', '--url', type=str, metavar="", help='url (with subdir/subdomain) to download')
         exclusive_required = required.add_mutually_exclusive_group(required=True)
@@ -32,16 +31,17 @@ class Arguments:
         optional.add_argument('--limit', type=int, nargs='?', const=True, metavar='int', help='limit the number of snapshots to download')
         
         behavior = parser.add_argument_group('manipulate behavior')
-        behavior.add_argument('-o', '--output', type=str, metavar="", help='output folder - defaults to current directory')
+        behavior.add_argument('-o', '--output', type=str, metavar="", help='output for all files - defaults to current directory')
+        behavior.add_argument('-m', '--metadata', type=str, metavar="", help='change directory for db/cdx/csv/log files')
         behavior.add_argument('--log', action='store_true', help='save a log file into the output folder')
         behavior.add_argument('--progress', action='store_true', help='show a progress bar')
         behavior.add_argument('--no-redirect', action='store_true', help='do not follow redirects by archive.org')
-        #behavior.add_argument('--verbosity', type=str, default="info", metavar="", help='verbosity level (info, trace)')
         behavior.add_argument('--retry', type=int, default=0, metavar="", help='retry failed downloads (opt tries as int, else infinite)')
         behavior.add_argument('--workers', type=int, default=1, metavar="", help='number of workers (simultaneous downloads)')
         # behavior.add_argument('--convert-links', action='store_true', help='Convert all links in the files to local paths. Requires -c/--current')
         behavior.add_argument('--delay', type=int, default=0, metavar="", help='delay between each download in seconds')
-        
+        behavior.add_argument('--verbose', action='store_true', help='overwritten by progress - gives detailed output')
+
         special = parser.add_argument_group('special')
         special.add_argument('--reset', action='store_true', help='reset the job and ignore existing cdx/db/csv files')
         special.add_argument('--keep', action='store_true', help='keep all files after the job finished')
@@ -75,11 +75,11 @@ class Configuration:
         
         if cls.output is None:
             cls.output = os.path.join(os.getcwd(), "waybackup_snapshots")
+        if cls.metadata is None:
+            cls.metadata = cls.output
         os.makedirs(cls.output, exist_ok=True) if not cls.save else None
-        
-        if cls.log is True:
-            cls.log = os.path.join(cls.output, f"waybackup_{sanitize_filename(cls.url)}.log")
-        
+        os.makedirs(cls.metadata, exist_ok=True) if not cls.save else None
+                
         if cls.all:
             cls.mode = "all"
         if cls.last:
@@ -91,10 +91,13 @@ class Configuration:
         
         if cls.filetype:
             cls.filetype = [ft.lower().strip() for ft in cls.filetype.split(",")]
-        
-        cls.cdxfile = os.path.join(cls.output, f"waybackup_{sanitize_filename(cls.url)}.cdx")
-        cls.dbfile = os.path.join(cls.output, f"waybackup_{sanitize_filename(cls.url)}.db")
-        cls.csvfile = os.path.join(cls.output, f"waybackup_{sanitize_filename(cls.url)}.csv")
+
+        base_path = cls.metadata
+        base_name = f"waybackup_{sanitize_filename(cls.url)}"
+        cls.cdxfile = os.path.join(base_path, f"{base_name}.cdx")
+        cls.dbfile = os.path.join(base_path, f"{base_name}.db")
+        cls.csvfile = os.path.join(base_path, f"{base_name}.csv")
+        cls.log = os.path.join(base_path, f"{base_name}.log") if cls.log else None
         
         if cls.reset:
             os.remove(cls.cdxfile) if os.path.isfile(cls.cdxfile) else None

{pywaybackup-3.1.0 → pywaybackup-3.2.1}/pywaybackup/Converter.py

@@ -29,7 +29,7 @@ class Converter:
 
 
     @classmethod
-    def links(cls, filepath, status_message=None):
+    def links(cls, filepath, status_content=None):
         """
         Convert all links in a HTML / CSS / JS file to local paths.
         """
@@ -72,7 +72,7 @@ class Converter:
             if original_url.startswith("//"):
                 external = True
             if external:
-                status_message.trace(status="", type=f"{count}/{len(links)}", message="External url")
+                status_message.trace(status="", info=f"{count}/{len(links)}", content="External url")
                 return original_url
 
             # convert the url to a relative path to the local root (download dir) if it's a valid path, else return the original url
@@ -87,7 +87,7 @@ class Converter:
                 if original_url.startswith("../"): # if file is already ../ check if it's not too many steps up
                     original_url = f"{cls.define_root_steps(filepath)}{original_url.split('../')[-1].lstrip('/')}"
             else:
-                status_message.trace(status="", type="", message=f"{count}/{len(links)}: URL is not a valid path")
+                status_message.trace(status="", info="", content=f"{count}/{len(links)}: URL is not a valid path")
 
             return original_url
 
@@ -158,24 +158,24 @@ class Converter:
 
         if os.path.isfile(filepath):
             if magic.from_file(filepath, mime=True).split("/")[1] == "javascript":
-                status_message.trace(status="Error", type="", message="JS-file is not supported")
+                status_message.trace(status="Error", info="", content="JS-file is not supported")
                 return
             try:
-                with open(filepath, "r") as file:
+                with open(filepath, "r", encoding="utf-8") as file:
                     domain = config.domain
                     content = file.read()
                     links = extract_urls(content)
-                    status_message.store(message=f"\n-----> Convert: [{len(links)}] links in file")
+                    status_message.store(verbose=True, content=f"\n-----> Convert: [{len(links)}] links in file")
                     count = 1
                     for original_link in links:
-                        status_message.trace(status="ORIG", type=f"{count}/{len(links)}", message=original_link)
+                        status_message.trace(status="ORIG", info=f"{count}/{len(links)}", content=original_link)
                         new_link = local_url(original_link, domain, count)
                         if new_link != original_link:
-                            status_message.trace(status="CONV", type=f"{count}/{len(links)}", message=new_link)
+                            status_message.trace(status="CONV", info=f"{count}/{len(links)}", content=new_link)
                         content = content.replace(original_link, new_link)
                         count += 1
-                    file = open(filepath, "w")
+                    file = open(filepath, "w", encoding="utf-8")
                     file.write(content)
                     file.close()
             except UnicodeDecodeError:
-                status_message.trace(status="Error", type="", message="Could not decode file to convert links")
+                status_message.trace(status="Error", info="", content="Could not decode file to convert links")