pywaybackup 1.3.0__tar.gz → 1.3.2__tar.gz

{pywaybackup-1.3.0/pywaybackup.egg-info → pywaybackup-1.3.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: pywaybackup
-Version: 1.3.0
+Version: 1.3.2
 Summary: Download snapshots from the Wayback Machine
 Home-page: https://github.com/bitdruid/python-wayback-machine-downloader
 Author: bitdruid
@@ -27,10 +27,6 @@ Internet-archive is a nice source for several OSINT-information. This tool is a
 
 This tool allows you to download content from the Wayback Machine (archive.org). You can use it to download either the latest version or all versions of web page snapshots within a specified range.
 
-## Info
-
-Linux recommended: On windows machines, the path length is limited. It can only be overcome by editing the registry. Files which exceed the path length will not be downloaded.
-
 ## Installation
 
 ### Pip
@@ -48,6 +44,11 @@ Linux recommended: On windows machines, the path length is limited. It can only
    ```pip install .```
    - in a virtual env or use `--break-system-package`
 
+## Usage infos
+
+- Linux recommended: On Windows machines, the path length is limited. This can only be overcome by editing the registry. Files that exceed the path length will not be downloaded.
+- If you query an explicit file (e.g. a query-string `?query=this` or `login.html`), the `--explicit`-argument is recommended as a wildcard query may lead to an empty result.
+
 ## Arguments
 
 - `-h`, `--help`: Show the help message and exit.
@@ -55,25 +56,35 @@ Linux recommended: On windows machines, the path length is limited. It can only
 
 ### Required
 
-- `-u`, `--url`: The URL of the web page to download. This argument is required.
+- **`-u`**, **`--url`**:<br>
+  The URL of the web page to download. This argument is required.
 
 #### Mode Selection (Choose One)
-- `-c`, `--current`: Download the latest version of each file snapshot. You will get a rebuild of the current website with all available files (but not any original state because new and old versions are mixed).
-- `-f`, `--full`: Download snapshots of all timestamps. You will get a folder per timestamp with the files available at that time.
-- `-s`, `--save`: Save a page to the Wayback Machine. (beta)
+- **`-c`**, **`--current`**:<br>
+  Download the latest version of each file snapshot. You will get a rebuild of the current website with all available files (but not any original state because new and old versions are mixed).
+- **`-f`**, **`--full`**:<br>
+  Download snapshots of all timestamps. You will get a folder per timestamp with the files available at that time.
+- **`-s`**, **`--save`**:<br>
+  Save a page to the Wayback Machine. (beta)
 
 ### Optional query parameters
 
-- `-l`, `--list`: Only print the snapshots available within the specified range. Does not download the snapshots.
-- `-e`, `--explicit`: Only download the explicit given url. No wildcard subdomains or paths. Use e.g. to get root-only snapshots.
-- `-o`, `--output`: Defaults to `waybackup_snapshots` in the current directory. The folder where downloaded files will be saved.
+- **`-l`**, **`--list`**:<br>
+  Only print the snapshots available within the specified range. Does not download the snapshots.
+- **`-e`**, **`--explicit`**:<br>
+  Only download the explicit given URL. No wildcard subdomains or paths. Use e.g. to get root-only snapshots. This is recommended for explicit files like `login.html` or `?query=this`.
+- **`-o`**, **`--output`**:<br>
+  Defaults to `waybackup_snapshots` in the current directory. The folder where downloaded files will be saved.
 
 - **Range Selection:**<br>
-Specify the range in years or a specific timestamp either start, end or both. If you specify the `range` argument, the `start` and `end` arguments will be ignored. Format for timestamps: YYYYMMDDhhmmss. You can only give a year or increase specificity by going through the timestamp starting on the left.<br>
-(year 2019, year+month 201901, year+month+day 20190101, year+month+day+hour 2019010112)
-   - `-r`, `--range`: Specify the range in years for which to search and download snapshots.
-   - `--start`: Timestamp to start searching.
-   - `--end`: Timestamp to end searching.
+  Specify the range in years or a specific timestamp either start, end, or both. If you specify the `range` argument, the `start` and `end` arguments will be ignored. Format for timestamps: YYYYMMDDhhmmss. You can only give a year or increase specificity by going through the timestamp starting on the left.<br>
+  (year 2019, year+month 201901, year+month+day 20190101, year+month+day+hour 2019010112)
+   - **`-r`**, **`--range`**:<br>
+     Specify the range in years for which to search and download snapshots.
+   - **`--start`**:<br>
+     Timestamp to start searching.
+   - **`--end`**:<br>
+     Timestamp to end searching.
 
 ### Additional behavior manipulation
   
@@ -81,7 +92,7 @@ Specify the range in years or a specific timestamp either start, end or both. If
 Path defaults to output-dir. Saves a CSV file with the json-response for successfull downloads. If `--list` is set, the CSV contains the CDX list of snapshots. If `--current` or `--full` is set, CSV contains downloaded files. Named as `waybackup_<sanitized_url>.csv`.
 
 - **`--skip`** `<path>`:<br>
-Path defaults to output-dir. Checks for an existing `waybackup_<domain>.csv` for URLs to skip downloading. Useful for interrupted downloads. Files are checked by their root-domain, ensuring consistency across queries. This means that if you download `http://example.com/subdir1/` and later `http://example.com`, the second query will skip the first path.
+Path defaults to output-dir. Checks for an existing `waybackup_<sanitized_url>.csv` for URLs to skip downloading. Useful for interrupted downloads. Files are checked by their root-domain, ensuring consistency across queries. This means that if you download `http://example.com/subdir1/` and later `http://example.com`, the second query will skip the first path.
   
 - **`--no-redirect`**:<br>
 Disables following redirects of snapshots. Useful for preventing timestamp-folder mismatches caused by Archive.org redirects.

{pywaybackup-1.3.0 → pywaybackup-1.3.2}/README.md

@@ -11,10 +11,6 @@ Internet-archive is a nice source for several OSINT-information. This tool is a
 
 This tool allows you to download content from the Wayback Machine (archive.org). You can use it to download either the latest version or all versions of web page snapshots within a specified range.
 
-## Info
-
-Linux recommended: On windows machines, the path length is limited. It can only be overcome by editing the registry. Files which exceed the path length will not be downloaded.
-
 ## Installation
 
 ### Pip
@@ -32,6 +28,11 @@ Linux recommended: On windows machines, the path length is limited. It can only
    ```pip install .```
    - in a virtual env or use `--break-system-package`
 
+## Usage infos
+
+- Linux recommended: On Windows machines, the path length is limited. This can only be overcome by editing the registry. Files that exceed the path length will not be downloaded.
+- If you query an explicit file (e.g. a query-string `?query=this` or `login.html`), the `--explicit`-argument is recommended as a wildcard query may lead to an empty result.
+
 ## Arguments
 
 - `-h`, `--help`: Show the help message and exit.
@@ -39,25 +40,35 @@ Linux recommended: On windows machines, the path length is limited. It can only
 
 ### Required
 
-- `-u`, `--url`: The URL of the web page to download. This argument is required.
+- **`-u`**, **`--url`**:<br>
+  The URL of the web page to download. This argument is required.
 
 #### Mode Selection (Choose One)
-- `-c`, `--current`: Download the latest version of each file snapshot. You will get a rebuild of the current website with all available files (but not any original state because new and old versions are mixed).
-- `-f`, `--full`: Download snapshots of all timestamps. You will get a folder per timestamp with the files available at that time.
-- `-s`, `--save`: Save a page to the Wayback Machine. (beta)
+- **`-c`**, **`--current`**:<br>
+  Download the latest version of each file snapshot. You will get a rebuild of the current website with all available files (but not any original state because new and old versions are mixed).
+- **`-f`**, **`--full`**:<br>
+  Download snapshots of all timestamps. You will get a folder per timestamp with the files available at that time.
+- **`-s`**, **`--save`**:<br>
+  Save a page to the Wayback Machine. (beta)
 
 ### Optional query parameters
 
-- `-l`, `--list`: Only print the snapshots available within the specified range. Does not download the snapshots.
-- `-e`, `--explicit`: Only download the explicit given url. No wildcard subdomains or paths. Use e.g. to get root-only snapshots.
-- `-o`, `--output`: Defaults to `waybackup_snapshots` in the current directory. The folder where downloaded files will be saved.
+- **`-l`**, **`--list`**:<br>
+  Only print the snapshots available within the specified range. Does not download the snapshots.
+- **`-e`**, **`--explicit`**:<br>
+  Only download the explicit given URL. No wildcard subdomains or paths. Use e.g. to get root-only snapshots. This is recommended for explicit files like `login.html` or `?query=this`.
+- **`-o`**, **`--output`**:<br>
+  Defaults to `waybackup_snapshots` in the current directory. The folder where downloaded files will be saved.
 
 - **Range Selection:**<br>
-Specify the range in years or a specific timestamp either start, end or both. If you specify the `range` argument, the `start` and `end` arguments will be ignored. Format for timestamps: YYYYMMDDhhmmss. You can only give a year or increase specificity by going through the timestamp starting on the left.<br>
-(year 2019, year+month 201901, year+month+day 20190101, year+month+day+hour 2019010112)
-   - `-r`, `--range`: Specify the range in years for which to search and download snapshots.
-   - `--start`: Timestamp to start searching.
-   - `--end`: Timestamp to end searching.
+  Specify the range in years or a specific timestamp either start, end, or both. If you specify the `range` argument, the `start` and `end` arguments will be ignored. Format for timestamps: YYYYMMDDhhmmss. You can only give a year or increase specificity by going through the timestamp starting on the left.<br>
+  (year 2019, year+month 201901, year+month+day 20190101, year+month+day+hour 2019010112)
+   - **`-r`**, **`--range`**:<br>
+     Specify the range in years for which to search and download snapshots.
+   - **`--start`**:<br>
+     Timestamp to start searching.
+   - **`--end`**:<br>
+     Timestamp to end searching.
 
 ### Additional behavior manipulation
   
@@ -65,7 +76,7 @@ Specify the range in years or a specific timestamp either start, end or both. If
 Path defaults to output-dir. Saves a CSV file with the json-response for successfull downloads. If `--list` is set, the CSV contains the CDX list of snapshots. If `--current` or `--full` is set, CSV contains downloaded files. Named as `waybackup_<sanitized_url>.csv`.
 
 - **`--skip`** `<path>`:<br>
-Path defaults to output-dir. Checks for an existing `waybackup_<domain>.csv` for URLs to skip downloading. Useful for interrupted downloads. Files are checked by their root-domain, ensuring consistency across queries. This means that if you download `http://example.com/subdir1/` and later `http://example.com`, the second query will skip the first path.
+Path defaults to output-dir. Checks for an existing `waybackup_<sanitized_url>.csv` for URLs to skip downloading. Useful for interrupted downloads. Files are checked by their root-domain, ensuring consistency across queries. This means that if you download `http://example.com/subdir1/` and later `http://example.com`, the second query will skip the first path.
   
 - **`--no-redirect`**:<br>
 Disables following redirects of snapshots. Useful for preventing timestamp-folder mismatches caused by Archive.org redirects.

pywaybackup-1.3.2/pywaybackup/__version__.py

@@ -0,0 +1 @@
+__version__ = "1.3.2"

{pywaybackup-1.3.0 → pywaybackup-1.3.2}/pywaybackup/archive.py

@@ -97,7 +97,7 @@ def print_list():
 # create filelist
 # timestamp format yyyyMMddhhmmss
 def query_list(url: str, range: int, start: int, end: int, explicit: bool, mode: str, cdxbackup: str, cdxinject: str):
-
+    
     def inject(cdxinject):
         if os.path.isfile(cdxinject):
             vb.write("\nInjecting CDX data...")
@@ -121,13 +121,15 @@ def query_list(url: str, range: int, start: int, end: int, explicit: bool, mode:
 
         domain, subdir, filename = url_split(url)
         if domain and not subdir and not filename:
-            cdx_url = f"*.{domain}/*" if not explicit else f"{domain}"
+            cdx_url = f"{domain}"
         if domain and subdir and not filename:
-            cdx_url = f"{domain}/{subdir}/*"
+            cdx_url = f"{domain}/{subdir}"
         if domain and subdir and filename:
-            cdx_url = f"{domain}/{subdir}/{filename}/*"
+            cdx_url = f"{domain}/{subdir}/{filename}"
         if domain and not subdir and filename:
-            cdx_url = f"{domain}/{filename}/*"
+            cdx_url = f"{domain}/{filename}"
+        if not explicit:
+            cdx_url = f"{cdx_url}/*"
 
         vb.write(f"---> {cdx_url}")
         cdxQuery = f"https://web.archive.org/cdx/search/cdx?output=json&url={cdx_url}{query_range}&fl=timestamp,digest,mimetype,statuscode,original&filter!=statuscode:200"
@@ -146,6 +148,7 @@ def query_list(url: str, range: int, start: int, end: int, explicit: bool, mode:
 
         return cdxResult
 
+    cdxResult = None
     if cdxinject:
         cdxResult = inject(cdxinject)
     if not cdxResult:

{pywaybackup-1.3.0 → pywaybackup-1.3.2/pywaybackup.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: pywaybackup
-Version: 1.3.0
+Version: 1.3.2
 Summary: Download snapshots from the Wayback Machine
 Home-page: https://github.com/bitdruid/python-wayback-machine-downloader
 Author: bitdruid
@@ -27,10 +27,6 @@ Internet-archive is a nice source for several OSINT-information. This tool is a
 
 This tool allows you to download content from the Wayback Machine (archive.org). You can use it to download either the latest version or all versions of web page snapshots within a specified range.
 
-## Info
-
-Linux recommended: On windows machines, the path length is limited. It can only be overcome by editing the registry. Files which exceed the path length will not be downloaded.
-
 ## Installation
 
 ### Pip
@@ -48,6 +44,11 @@ Linux recommended: On windows machines, the path length is limited. It can only
    ```pip install .```
    - in a virtual env or use `--break-system-package`
 
+## Usage infos
+
+- Linux recommended: On Windows machines, the path length is limited. This can only be overcome by editing the registry. Files that exceed the path length will not be downloaded.
+- If you query an explicit file (e.g. a query-string `?query=this` or `login.html`), the `--explicit`-argument is recommended as a wildcard query may lead to an empty result.
+
 ## Arguments
 
 - `-h`, `--help`: Show the help message and exit.
@@ -55,25 +56,35 @@ Linux recommended: On windows machines, the path length is limited. It can only
 
 ### Required
 
-- `-u`, `--url`: The URL of the web page to download. This argument is required.
+- **`-u`**, **`--url`**:<br>
+  The URL of the web page to download. This argument is required.
 
 #### Mode Selection (Choose One)
-- `-c`, `--current`: Download the latest version of each file snapshot. You will get a rebuild of the current website with all available files (but not any original state because new and old versions are mixed).
-- `-f`, `--full`: Download snapshots of all timestamps. You will get a folder per timestamp with the files available at that time.
-- `-s`, `--save`: Save a page to the Wayback Machine. (beta)
+- **`-c`**, **`--current`**:<br>
+  Download the latest version of each file snapshot. You will get a rebuild of the current website with all available files (but not any original state because new and old versions are mixed).
+- **`-f`**, **`--full`**:<br>
+  Download snapshots of all timestamps. You will get a folder per timestamp with the files available at that time.
+- **`-s`**, **`--save`**:<br>
+  Save a page to the Wayback Machine. (beta)
 
 ### Optional query parameters
 
-- `-l`, `--list`: Only print the snapshots available within the specified range. Does not download the snapshots.
-- `-e`, `--explicit`: Only download the explicit given url. No wildcard subdomains or paths. Use e.g. to get root-only snapshots.
-- `-o`, `--output`: Defaults to `waybackup_snapshots` in the current directory. The folder where downloaded files will be saved.
+- **`-l`**, **`--list`**:<br>
+  Only print the snapshots available within the specified range. Does not download the snapshots.
+- **`-e`**, **`--explicit`**:<br>
+  Only download the explicit given URL. No wildcard subdomains or paths. Use e.g. to get root-only snapshots. This is recommended for explicit files like `login.html` or `?query=this`.
+- **`-o`**, **`--output`**:<br>
+  Defaults to `waybackup_snapshots` in the current directory. The folder where downloaded files will be saved.
 
 - **Range Selection:**<br>
-Specify the range in years or a specific timestamp either start, end or both. If you specify the `range` argument, the `start` and `end` arguments will be ignored. Format for timestamps: YYYYMMDDhhmmss. You can only give a year or increase specificity by going through the timestamp starting on the left.<br>
-(year 2019, year+month 201901, year+month+day 20190101, year+month+day+hour 2019010112)
-   - `-r`, `--range`: Specify the range in years for which to search and download snapshots.
-   - `--start`: Timestamp to start searching.
-   - `--end`: Timestamp to end searching.
+  Specify the range in years or a specific timestamp either start, end, or both. If you specify the `range` argument, the `start` and `end` arguments will be ignored. Format for timestamps: YYYYMMDDhhmmss. You can only give a year or increase specificity by going through the timestamp starting on the left.<br>
+  (year 2019, year+month 201901, year+month+day 20190101, year+month+day+hour 2019010112)
+   - **`-r`**, **`--range`**:<br>
+     Specify the range in years for which to search and download snapshots.
+   - **`--start`**:<br>
+     Timestamp to start searching.
+   - **`--end`**:<br>
+     Timestamp to end searching.
 
 ### Additional behavior manipulation
   
@@ -81,7 +92,7 @@ Specify the range in years or a specific timestamp either start, end or both. If
 Path defaults to output-dir. Saves a CSV file with the json-response for successfull downloads. If `--list` is set, the CSV contains the CDX list of snapshots. If `--current` or `--full` is set, CSV contains downloaded files. Named as `waybackup_<sanitized_url>.csv`.
 
 - **`--skip`** `<path>`:<br>
-Path defaults to output-dir. Checks for an existing `waybackup_<domain>.csv` for URLs to skip downloading. Useful for interrupted downloads. Files are checked by their root-domain, ensuring consistency across queries. This means that if you download `http://example.com/subdir1/` and later `http://example.com`, the second query will skip the first path.
+Path defaults to output-dir. Checks for an existing `waybackup_<sanitized_url>.csv` for URLs to skip downloading. Useful for interrupted downloads. Files are checked by their root-domain, ensuring consistency across queries. This means that if you download `http://example.com/subdir1/` and later `http://example.com`, the second query will skip the first path.
   
 - **`--no-redirect`**:<br>
 Disables following redirects of snapshots. Useful for preventing timestamp-folder mismatches caused by Archive.org redirects.

pywaybackup-1.3.0/pywaybackup/__version__.py

@@ -1 +0,0 @@
-__version__ = "1.3.0"