pywaybackup 0.8.1__tar.gz → 1.0.2__tar.gz

pywaybackup-1.0.2/PKG-INFO

@@ -0,0 +1,187 @@
+Metadata-Version: 2.1
+Name: pywaybackup
+Version: 1.0.2
+Summary: Download snapshots from the Wayback Machine
+Home-page: https://github.com/bitdruid/python-wayback-machine-downloader
+Author: bitdruid
+Author-email: bitdruid@outlook.com
+License: MIT
+Keywords: wayback machine internet archive
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: requests==2.31.0
+Requires-Dist: tqdm==4.66.2
+
+# archive wayback downloader
+
+[![PyPI](https://img.shields.io/pypi/v/pywaybackup)](https://pypi.org/project/pywaybackup/)
+[![PyPI - Downloads](https://img.shields.io/pypi/dm/pywaybackup)](https://pypi.org/project/pywaybackup/)
+![Python Version](https://img.shields.io/badge/Python-3.6-blue)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+
+Downloading archived web pages from the [Wayback Machine](https://archive.org/web/).
+
+Internet-archive is a nice source for several OSINT-information. This script is a work in progress to query and fetch archived web pages.
+
+## Installation
+
+### Pip
+
+1. Install the package <br>
+   ```pip install pywaybackup```
+2. Run the script <br>
+   ```waybackup -h```
+
+### Manual
+
+1. Clone the repository <br>
+   ```git clone https://github.com/bitdruid/python-wayback-machine-downloader.git```
+2. Install <br>
+   ```pip install .```
+   - in a virtual env or use `--break-system-package`
+
+## Usage
+
+This script allows you to download content from the Wayback Machine (archive.org). You can use it to download either the latest version or all versions of web page snapshots within a specified range.
+
+### Arguments
+
+- `-h`, `--help`: Show the help message and exit.
+- `-a`, `--about`: Show information about the script and exit.
+
+#### Required Arguments
+
+- `-u`, `--url`: The URL of the web page to download. This argument is required.
+
+#### Mode Selection (Choose One)
+
+- `-c`, `--current`: Download the latest version of each file snapshot. You will get a rebuild of the current website with all available files (but not any original state because new and old versions are mixed).
+- `-f`, `--full`: Download snapshots of all timestamps. You will get a folder per timestamp with the files available at that time.
+- `-s`, `--save`: Save a page to the Wayback Machine. (beta)
+
+#### Optional Arguments
+
+- `-l`, `--list`: Only print the snapshots available within the specified range. Does not download the snapshots.
+- `-e`, `--explicit`: Only download the explicit given url. No wildcard subdomains or paths. Use e.g. to get root-only snapshots.
+- `-o`, `--output`: The folder where downloaded files will be saved.
+
+- **Range Selection:**<br>
+Specify the range in years or a specific timestamp either start, end or both. If you specify the `range` argument, the `start` and `end` arguments will be ignored. Format for timestamps: YYYYMMDDhhmmss. You can only give a year or increase specificity by going through the timestamp starting on the left.<br>
+(year 2019, year+month 201901, year+month+day 20190101, year+month+day+hour 2019010112)
+   - `-r`, `--range`: Specify the range in years for which to search and download snapshots.
+   - `--start`: Timestamp to start searching.
+   - `--end`: Timestamp to end searching.
+
+#### Additional
+
+- `--csv`: Save a csv file with the list of snapshots inside the output folder or a specified folder. If you set `--list` the csv will contain the cdx list of snapshots. If you set either `--current` or `--full` the csv will contain the downloaded files.
+- `--no-redirect`: Do not follow redirects of snapshots. Archive.org sometimes redirects to a different snapshot for several reasons. Downloading redirects may lead to timestamp-folders which contain some files with a different timestamp. This does not matter if you only want to download the latest version (`-c`).
+- `--verbosity`: Set the verbosity: json (print json response), progress (show progress bar).
+- `--retry`: Retry failed downloads. You can specify the number of retry attempts as an integer.
+- `--workers`: The number of workers to use for downloading (simultaneous downloads). Default is 1. A safe spot is about 10 workers. Beware: Using too many workers will lead into refused connections from the Wayback Machine. Duration about 1.5 minutes.
+
+### Examples
+
+Download latest snapshot of all files:<br>
+`waybackup -u http://example.com -c`
+
+Download latest snapshot of a specific file:<br>
+`waybackup -u http://example.com/subdir/file.html -c`
+
+Download all snapshots sorted per timestamp with a specified range and do not follow redirects:<br>
+`waybackup -u http://example.com -f -r 5 --no-redirect`
+
+Download all snapshots sorted per timestamp with a specified range and save to a specified folder with 3 workers:<br>
+`waybackup -u http://example.com -f -r 5 -o /home/user/Downloads/snapshots --workers 3`
+
+Download all snapshots from 2020 to 12th of December 2022 with 4 workers, save a csv and show a progress bar:
+`waybackup -u http://example.com -f --start 2020 --end 20221212 --workers 4 --csv --verbosity progress`
+
+Download all snapshots and output a json response:<br>
+`waybackup -u http://example.com -f --verbosity json`
+
+List available snapshots per timestamp without downloading and save a csv file to home folder:<br>
+`waybackup -u http://example.com -f -l --csv /home/user/Downloads`
+
+## Output path structure
+
+The output path is currently structured as follows by an example for the query:<br>
+`http://example.com/subdir1/subdir2/assets/`:
+<br><br>
+For the current version (`-c`):
+- The requested path will only include all files/folders starting from your query-path.
+```
+your/path/waybackup_snapshots/
+└── the_root_of_your_query/ (example.com/)
+    └── subdir1/
+        └── subdir2/
+            └── assets/
+                ├── image.jpg
+                ├── style.css
+                ...
+```
+For all versions (`-f`):
+- Will currently create a folder named as the root of your query. Inside this folder, you will find all timestamps and per timestamp the path you requested.
+```
+your/path/waybackup_snapshots/
+└── the_root_of_your_query/ (example.com/)
+    ├── yyyymmddhhmmss/
+    │   ├── subidr1/
+    │   │   └── subdir2/
+    │   │       └── assets/
+    │   │           ├── image.jpg
+    │   │           └── style.css
+    ├── yyyymmddhhmmss/
+    │   ├── subdir1/
+    │   │   └── subdir2/
+    │   │       └── assets/
+    │   │           ├── image.jpg
+    │   │           └── style.css
+    ...
+```
+
+
+### Json Response
+
+For download queries:
+
+```
+[
+   {
+      "file": "/your/path/waybackup_snapshots/example.com/yyyymmddhhmmss/index.html",
+      "id": 1,
+      "redirect_timestamp": "yyyymmddhhmmss",
+      "redirect_url": "http://web.archive.org/web/yyyymmddhhmmssid_/http://example.com/",
+      "response": 200,
+      "timestamp": "yyyymmddhhmmss",
+      "url_archive": "http://web.archive.org/web/yyyymmddhhmmssid_/http://example.com/",
+      "url_origin": "http://example.com/"
+   },
+    ...
+]
+```
+
+For list queries:
+
+```
+[
+   {
+      "digest": "DIGESTOFSNAPSHOT",
+      "id": 1,
+      "mimetype": "text/html",
+      "status": "200",
+      "timestamp": "yyyymmddhhmmss",
+      "url": "http://example.com/"
+   },
+   ...
+]
+```
+
+## CSV Output
+
+The csv contains the json response in a table format.
+
+## Contributing
+
+I'm always happy for some feature requests to improve the usability of this script.
+Feel free to give suggestions and report issues. Project is still far from being perfect.

pywaybackup-1.0.2/README.md

@@ -0,0 +1,173 @@
+# archive wayback downloader
+
+[![PyPI](https://img.shields.io/pypi/v/pywaybackup)](https://pypi.org/project/pywaybackup/)
+[![PyPI - Downloads](https://img.shields.io/pypi/dm/pywaybackup)](https://pypi.org/project/pywaybackup/)
+![Python Version](https://img.shields.io/badge/Python-3.6-blue)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+
+Downloading archived web pages from the [Wayback Machine](https://archive.org/web/).
+
+Internet-archive is a nice source for several OSINT-information. This script is a work in progress to query and fetch archived web pages.
+
+## Installation
+
+### Pip
+
+1. Install the package <br>
+   ```pip install pywaybackup```
+2. Run the script <br>
+   ```waybackup -h```
+
+### Manual
+
+1. Clone the repository <br>
+   ```git clone https://github.com/bitdruid/python-wayback-machine-downloader.git```
+2. Install <br>
+   ```pip install .```
+   - in a virtual env or use `--break-system-package`
+
+## Usage
+
+This script allows you to download content from the Wayback Machine (archive.org). You can use it to download either the latest version or all versions of web page snapshots within a specified range.
+
+### Arguments
+
+- `-h`, `--help`: Show the help message and exit.
+- `-a`, `--about`: Show information about the script and exit.
+
+#### Required Arguments
+
+- `-u`, `--url`: The URL of the web page to download. This argument is required.
+
+#### Mode Selection (Choose One)
+
+- `-c`, `--current`: Download the latest version of each file snapshot. You will get a rebuild of the current website with all available files (but not any original state because new and old versions are mixed).
+- `-f`, `--full`: Download snapshots of all timestamps. You will get a folder per timestamp with the files available at that time.
+- `-s`, `--save`: Save a page to the Wayback Machine. (beta)
+
+#### Optional Arguments
+
+- `-l`, `--list`: Only print the snapshots available within the specified range. Does not download the snapshots.
+- `-e`, `--explicit`: Only download the explicit given url. No wildcard subdomains or paths. Use e.g. to get root-only snapshots.
+- `-o`, `--output`: The folder where downloaded files will be saved.
+
+- **Range Selection:**<br>
+Specify the range in years or a specific timestamp either start, end or both. If you specify the `range` argument, the `start` and `end` arguments will be ignored. Format for timestamps: YYYYMMDDhhmmss. You can only give a year or increase specificity by going through the timestamp starting on the left.<br>
+(year 2019, year+month 201901, year+month+day 20190101, year+month+day+hour 2019010112)
+   - `-r`, `--range`: Specify the range in years for which to search and download snapshots.
+   - `--start`: Timestamp to start searching.
+   - `--end`: Timestamp to end searching.
+
+#### Additional
+
+- `--csv`: Save a csv file with the list of snapshots inside the output folder or a specified folder. If you set `--list` the csv will contain the cdx list of snapshots. If you set either `--current` or `--full` the csv will contain the downloaded files.
+- `--no-redirect`: Do not follow redirects of snapshots. Archive.org sometimes redirects to a different snapshot for several reasons. Downloading redirects may lead to timestamp-folders which contain some files with a different timestamp. This does not matter if you only want to download the latest version (`-c`).
+- `--verbosity`: Set the verbosity: json (print json response), progress (show progress bar).
+- `--retry`: Retry failed downloads. You can specify the number of retry attempts as an integer.
+- `--workers`: The number of workers to use for downloading (simultaneous downloads). Default is 1. A safe spot is about 10 workers. Beware: Using too many workers will lead into refused connections from the Wayback Machine. Duration about 1.5 minutes.
+
+### Examples
+
+Download latest snapshot of all files:<br>
+`waybackup -u http://example.com -c`
+
+Download latest snapshot of a specific file:<br>
+`waybackup -u http://example.com/subdir/file.html -c`
+
+Download all snapshots sorted per timestamp with a specified range and do not follow redirects:<br>
+`waybackup -u http://example.com -f -r 5 --no-redirect`
+
+Download all snapshots sorted per timestamp with a specified range and save to a specified folder with 3 workers:<br>
+`waybackup -u http://example.com -f -r 5 -o /home/user/Downloads/snapshots --workers 3`
+
+Download all snapshots from 2020 to 12th of December 2022 with 4 workers, save a csv and show a progress bar:
+`waybackup -u http://example.com -f --start 2020 --end 20221212 --workers 4 --csv --verbosity progress`
+
+Download all snapshots and output a json response:<br>
+`waybackup -u http://example.com -f --verbosity json`
+
+List available snapshots per timestamp without downloading and save a csv file to home folder:<br>
+`waybackup -u http://example.com -f -l --csv /home/user/Downloads`
+
+## Output path structure
+
+The output path is currently structured as follows by an example for the query:<br>
+`http://example.com/subdir1/subdir2/assets/`:
+<br><br>
+For the current version (`-c`):
+- The requested path will only include all files/folders starting from your query-path.
+```
+your/path/waybackup_snapshots/
+└── the_root_of_your_query/ (example.com/)
+    └── subdir1/
+        └── subdir2/
+            └── assets/
+                ├── image.jpg
+                ├── style.css
+                ...
+```
+For all versions (`-f`):
+- Will currently create a folder named as the root of your query. Inside this folder, you will find all timestamps and per timestamp the path you requested.
+```
+your/path/waybackup_snapshots/
+└── the_root_of_your_query/ (example.com/)
+    ├── yyyymmddhhmmss/
+    │   ├── subidr1/
+    │   │   └── subdir2/
+    │   │       └── assets/
+    │   │           ├── image.jpg
+    │   │           └── style.css
+    ├── yyyymmddhhmmss/
+    │   ├── subdir1/
+    │   │   └── subdir2/
+    │   │       └── assets/
+    │   │           ├── image.jpg
+    │   │           └── style.css
+    ...
+```
+
+
+### Json Response
+
+For download queries:
+
+```
+[
+   {
+      "file": "/your/path/waybackup_snapshots/example.com/yyyymmddhhmmss/index.html",
+      "id": 1,
+      "redirect_timestamp": "yyyymmddhhmmss",
+      "redirect_url": "http://web.archive.org/web/yyyymmddhhmmssid_/http://example.com/",
+      "response": 200,
+      "timestamp": "yyyymmddhhmmss",
+      "url_archive": "http://web.archive.org/web/yyyymmddhhmmssid_/http://example.com/",
+      "url_origin": "http://example.com/"
+   },
+    ...
+]
+```
+
+For list queries:
+
+```
+[
+   {
+      "digest": "DIGESTOFSNAPSHOT",
+      "id": 1,
+      "mimetype": "text/html",
+      "status": "200",
+      "timestamp": "yyyymmddhhmmss",
+      "url": "http://example.com/"
+   },
+   ...
+]
+```
+
+## CSV Output
+
+The csv contains the json response in a table format.
+
+## Contributing
+
+I'm always happy for some feature requests to improve the usability of this script.
+Feel free to give suggestions and report issues. Project is still far from being perfect.

{pywaybackup-0.8.1 → pywaybackup-1.0.2}/pywaybackup/SnapshotCollection.py

@@ -36,11 +36,11 @@ class SnapshotCollection:
     @classmethod
     def create_collection(cls):
         new_collection = []
-        for cdx_entry in cls.SNAPSHOT_COLLECTION:
+        for idx, cdx_entry in enumerate(cls.SNAPSHOT_COLLECTION):
             timestamp, url = cdx_entry["timestamp"], cdx_entry["url"]
             url_archive = f"http://web.archive.org/web/{timestamp}{cls._url_get_filetype(url)}/{url}"
             collection_entry = {
-                "id": cls.SNAPSHOT_COLLECTION.index(cdx_entry),
+                "id": idx,
                 "timestamp": timestamp,
                 "url_archive": url_archive,
                 "url_origin": url,
@@ -65,7 +65,7 @@ class SnapshotCollection:
         - download_file: The output path for the snapshot entry (str) with filename.
         """
         timestamp, url = collection_entry["timestamp"], collection_entry["url_origin"]
-        domain, subdir, filename = cls._url_split(url)
+        domain, subdir, filename = cls.url_split(url, index=True)
         if cls.MODE_CURRENT:
             download_dir = os.path.join(output, domain, subdir)
         else:
@@ -109,12 +109,17 @@ class SnapshotCollection:
         return urltype
 
     @classmethod
-    def _url_split(cls, url):
+    def url_split(cls, url, index=False):
         """
         Split a URL into domain, subdir and filename.
         """
+        if not urlparse(url).scheme:
+            url = "http://" + url
         parsed_url = urlparse(url)
-        domain = parsed_url.netloc
-        subdir = parsed_url.path.strip("/").rsplit("/", 1)[0]
-        filename = parsed_url.path.split("/")[-1] or "index.html"
+        domain = parsed_url.netloc.split("@")[-1].split(":")[0] # split mailto: and port
+        filename = parsed_url.path.split("/")[-1]
+        if index is True and filename == "":
+            filename = "index.html"
+        subdir = parsed_url.path.strip("/").replace(filename, "").strip("/")
+        filename = filename.replace("%20", " ") # replace url encoded spaces
         return domain, subdir, filename

pywaybackup-1.0.2/pywaybackup/__version__.py

@@ -0,0 +1 @@
+__version__ = "1.0.2"

{pywaybackup-0.8.1 → pywaybackup-1.0.2}/pywaybackup/archive.py

@@ -89,12 +89,25 @@ def query_list(url: str, range: int, start: int, end: int, explicit: bool, mode:
     try:
         v.write("\nQuerying snapshots...")
         query_range = ""
+        
         if not range:
             if start: query_range = query_range + f"&from={start}"
             if end: query_range = query_range + f"&to={end}"
         else: 
             query_range = "&from=" + str(datetime.now().year - range)
-        cdx_url = f"*.{url}/*" if not explicit else f"{url}"
+
+        # parse user input url and create according cdx url
+        domain, subdir, filename = sc.url_split(url)
+        if domain and not subdir and not filename:
+            cdx_url = f"*.{domain}/*" if not explicit else f"{domain}"
+        if domain and subdir and not filename:
+            cdx_url = f"{domain}/{subdir}/*"
+        if domain and subdir and filename:
+            cdx_url = f"{domain}/{subdir}/{filename}/*"
+        if domain and not subdir and filename:
+            cdx_url = f"{domain}/{filename}/*"
+
+        print(f"---> {cdx_url}")
         cdxQuery = f"https://web.archive.org/cdx/search/xd?output=json&url={cdx_url}{query_range}&fl=timestamp,digest,mimetype,statuscode,original&filter!=statuscode:200"
         cdxResult = requests.get(cdxQuery)
         sc.create_list(cdxResult, mode)
@@ -107,7 +120,7 @@ def query_list(url: str, range: int, start: int, end: int, explicit: bool, mode:
 
 
 # example download: http://web.archive.org/web/20190815104545id_/https://www.google.com/
-def download_list(output, retry, no_redirect, worker):
+def download_list(output, retry, no_redirect, workers):
     """
     Download a list of urls in format: [{"timestamp": "20190815104545", "url": "https://www.google.com/"}]
     """
@@ -115,18 +128,19 @@ def download_list(output, retry, no_redirect, worker):
         v.write("\nNothing to download");
         return
     v.write("\nDownloading snapshots...", progress=0)
-    if worker > 1:
-        v.write(f"\n-----> Simultaneous downloads: {worker}")
-        batch_size = sc.count_list() // worker + 1
+    if workers > 1:
+        v.write(f"\n-----> Simultaneous downloads: {workers}")
+        batch_size = sc.count_list() // workers + 1
     else:
         batch_size = sc.count_list()
     sc.create_collection()
+    v.write("\n-----> Snapshots prepared")
     batch_list = [sc.SNAPSHOT_COLLECTION[i:i + batch_size] for i in range(0, len(sc.SNAPSHOT_COLLECTION), batch_size)]    
     threads = []
     worker = 0
     for batch in batch_list:
         worker += 1
-        thread = threading.Thread(target=download_loop, args=(batch, output, worker, retry, no_redirect))
+        thread = threading.Thread(target=download_loop, args=(batch, output, workers, retry, no_redirect))
         threads.append(thread)
         thread.start()
     for thread in threads:
@@ -256,15 +270,18 @@ def parse_response_code(response_code: int):
         return RESPONSE_CODE_DICT[response_code]
     return "Unknown response code"
 
-def save_csv(csv_path: str):
+def save_csv(csv_path: str, url: str):
     """
     Write a CSV file with the list of snapshots.
     """
     import csv
+    disallowed = ['<', '>', ':', '"', '/', '\\', '|', '?', '*']
+    for char in disallowed:
+        url = url.replace(char, '.')
     if sc.count_list() > 0:
         v.write("\nSaving CSV file...")
         os.makedirs(os.path.abspath(csv_path), exist_ok=True)
-        with open(os.path.join(csv_path, "waybackup.csv"), mode='w') as file:
+        with open(os.path.join(csv_path, f"waybackup_{url}.csv"), mode='w') as file:
             row = csv.DictWriter(file, sc.SNAPSHOT_COLLECTION[0].keys())
             row.writeheader()
             for snapshot in sc.SNAPSHOT_COLLECTION:

{pywaybackup-0.8.1 → pywaybackup-1.0.2}/pywaybackup/arguments.py

@@ -1,3 +1,4 @@
+import sys
 import argparse
 from pywaybackup.__version__ import __version__
 
@@ -7,7 +8,7 @@ def parse():
     parser.add_argument('-a', '--about', action='version', version='%(prog)s ' + __version__ + ' by @bitdruid -> https://github.com/bitdruid')
 
     required = parser.add_argument_group('required')
-    required.add_argument('-u', '--url', type=str, help='URL to use')
+    required.add_argument('-u', '--url', type=str, metavar="", help='URL to use')
     exclusive_required = required.add_mutually_exclusive_group(required=True)
     exclusive_required.add_argument('-c', '--current', action='store_true', help='Download the latest version of each file snapshot (opt range in y)')
     exclusive_required.add_argument('-f', '--full', action='store_true', help='Download snapshots of all timestamps (opt range in y)')
@@ -16,18 +17,18 @@ def parse():
     optional = parser.add_argument_group('optional')
     optional.add_argument('-l', '--list', action='store_true', help='Only print snapshots (opt range in y)')
     optional.add_argument('-e', '--explicit', action='store_true', help='Search only for the explicit given url')
-    optional.add_argument('-o', '--output', type=str, help='Output folder defaults to current directory')
-    optional.add_argument('-r', '--range', type=int, help='Range in years to search')
-    optional.add_argument('--start', type=int, help='Start timestamp format: YYYYMMDDhhmmss')
-    optional.add_argument('--end', type=int, help='End timestamp format: YYYYMMDDhhmmss')
+    optional.add_argument('-o', '--output', type=str, metavar="", help='Output folder defaults to current directory')
+    optional.add_argument('-r', '--range', type=int, metavar="", help='Range in years to search')
+    optional.add_argument('--start', type=int, metavar="", help='Start timestamp format: YYYYMMDDhhmmss')
+    optional.add_argument('--end', type=int, metavar="", help='End timestamp format: YYYYMMDDhhmmss')
 
     special = parser.add_argument_group('special')
-    special.add_argument('--csv', type=str, nargs='?', const=True, help='Save a csv file on a given path or defaults to the output folder')
+    special.add_argument('--csv', type=str, nargs='?', const=True, metavar='', help='Save a csv file on a given path or defaults to the output folder')
     special.add_argument('--no-redirect', action='store_true', help='Do not follow redirects by archive.org')
-    special.add_argument('--verbosity', type=str, default="standard", choices=["standard", "progress", "json"], help='Verbosity level')
-    special.add_argument('--retry', type=int, default=0, metavar="X-TIMES", help='Retry failed downloads (opt tries as int, else infinite)')
-    special.add_argument('--worker', type=int, default=1, metavar="AMOUNT", help='Number of worker (simultaneous downloads)')
+    special.add_argument('--verbosity', type=str, default="standard", metavar="", help='["progress", "json"] Verbosity level')
+    special.add_argument('--retry', type=int, default=0, metavar="", help='Retry failed downloads (opt tries as int, else infinite)')
+    special.add_argument('--workers', type=int, default=1, metavar="", help='Number of workers (simultaneous downloads)')
 
-    args = parser.parse_args()
+    args = parser.parse_args(args=None if sys.argv[1:] else ['--help']) # if no arguments are given, print help
 
     return args

{pywaybackup-0.8.1 → pywaybackup-1.0.2}/pywaybackup/main.py

@@ -25,9 +25,9 @@ def main():
         if args.list:
             archive.print_list()
         else:
-            archive.download_list(args.output, args.retry, args.no_redirect, args.worker)
+            archive.download_list(args.output, args.retry, args.no_redirect, args.workers)
         if args.csv:
-            archive.save_csv(args.csv)
+            archive.save_csv(args.csv, args.url)
     v.close()
 
 if __name__ == "__main__":

pywaybackup-1.0.2/pywaybackup.egg-info/PKG-INFO

@@ -0,0 +1,187 @@
+Metadata-Version: 2.1
+Name: pywaybackup
+Version: 1.0.2
+Summary: Download snapshots from the Wayback Machine
+Home-page: https://github.com/bitdruid/python-wayback-machine-downloader
+Author: bitdruid
+Author-email: bitdruid@outlook.com
+License: MIT
+Keywords: wayback machine internet archive
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: requests==2.31.0
+Requires-Dist: tqdm==4.66.2
+
+# archive wayback downloader
+
+[![PyPI](https://img.shields.io/pypi/v/pywaybackup)](https://pypi.org/project/pywaybackup/)
+[![PyPI - Downloads](https://img.shields.io/pypi/dm/pywaybackup)](https://pypi.org/project/pywaybackup/)
+![Python Version](https://img.shields.io/badge/Python-3.6-blue)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+
+Downloading archived web pages from the [Wayback Machine](https://archive.org/web/).
+
+Internet-archive is a nice source for several OSINT-information. This script is a work in progress to query and fetch archived web pages.
+
+## Installation
+
+### Pip
+
+1. Install the package <br>
+   ```pip install pywaybackup```
+2. Run the script <br>
+   ```waybackup -h```
+
+### Manual
+
+1. Clone the repository <br>
+   ```git clone https://github.com/bitdruid/python-wayback-machine-downloader.git```
+2. Install <br>
+   ```pip install .```
+   - in a virtual env or use `--break-system-package`
+
+## Usage
+
+This script allows you to download content from the Wayback Machine (archive.org). You can use it to download either the latest version or all versions of web page snapshots within a specified range.
+
+### Arguments
+
+- `-h`, `--help`: Show the help message and exit.
+- `-a`, `--about`: Show information about the script and exit.
+
+#### Required Arguments
+
+- `-u`, `--url`: The URL of the web page to download. This argument is required.
+
+#### Mode Selection (Choose One)
+
+- `-c`, `--current`: Download the latest version of each file snapshot. You will get a rebuild of the current website with all available files (but not any original state because new and old versions are mixed).
+- `-f`, `--full`: Download snapshots of all timestamps. You will get a folder per timestamp with the files available at that time.
+- `-s`, `--save`: Save a page to the Wayback Machine. (beta)
+
+#### Optional Arguments
+
+- `-l`, `--list`: Only print the snapshots available within the specified range. Does not download the snapshots.
+- `-e`, `--explicit`: Only download the explicit given url. No wildcard subdomains or paths. Use e.g. to get root-only snapshots.
+- `-o`, `--output`: The folder where downloaded files will be saved.
+
+- **Range Selection:**<br>
+Specify the range in years or a specific timestamp either start, end or both. If you specify the `range` argument, the `start` and `end` arguments will be ignored. Format for timestamps: YYYYMMDDhhmmss. You can only give a year or increase specificity by going through the timestamp starting on the left.<br>
+(year 2019, year+month 201901, year+month+day 20190101, year+month+day+hour 2019010112)
+   - `-r`, `--range`: Specify the range in years for which to search and download snapshots.
+   - `--start`: Timestamp to start searching.
+   - `--end`: Timestamp to end searching.
+
+#### Additional
+
+- `--csv`: Save a csv file with the list of snapshots inside the output folder or a specified folder. If you set `--list` the csv will contain the cdx list of snapshots. If you set either `--current` or `--full` the csv will contain the downloaded files.
+- `--no-redirect`: Do not follow redirects of snapshots. Archive.org sometimes redirects to a different snapshot for several reasons. Downloading redirects may lead to timestamp-folders which contain some files with a different timestamp. This does not matter if you only want to download the latest version (`-c`).
+- `--verbosity`: Set the verbosity: json (print json response), progress (show progress bar).
+- `--retry`: Retry failed downloads. You can specify the number of retry attempts as an integer.
+- `--workers`: The number of workers to use for downloading (simultaneous downloads). Default is 1. A safe spot is about 10 workers. Beware: Using too many workers will lead into refused connections from the Wayback Machine. Duration about 1.5 minutes.
+
+### Examples
+
+Download latest snapshot of all files:<br>
+`waybackup -u http://example.com -c`
+
+Download latest snapshot of a specific file:<br>
+`waybackup -u http://example.com/subdir/file.html -c`
+
+Download all snapshots sorted per timestamp with a specified range and do not follow redirects:<br>
+`waybackup -u http://example.com -f -r 5 --no-redirect`
+
+Download all snapshots sorted per timestamp with a specified range and save to a specified folder with 3 workers:<br>
+`waybackup -u http://example.com -f -r 5 -o /home/user/Downloads/snapshots --workers 3`
+
+Download all snapshots from 2020 to 12th of December 2022 with 4 workers, save a csv and show a progress bar:
+`waybackup -u http://example.com -f --start 2020 --end 20221212 --workers 4 --csv --verbosity progress`
+
+Download all snapshots and output a json response:<br>
+`waybackup -u http://example.com -f --verbosity json`
+
+List available snapshots per timestamp without downloading and save a csv file to home folder:<br>
+`waybackup -u http://example.com -f -l --csv /home/user/Downloads`
+
+## Output path structure
+
+The output path is currently structured as follows by an example for the query:<br>
+`http://example.com/subdir1/subdir2/assets/`:
+<br><br>
+For the current version (`-c`):
+- The requested path will only include all files/folders starting from your query-path.
+```
+your/path/waybackup_snapshots/
+└── the_root_of_your_query/ (example.com/)
+    └── subdir1/
+        └── subdir2/
+            └── assets/
+                ├── image.jpg
+                ├── style.css
+                ...
+```
+For all versions (`-f`):
+- Will currently create a folder named as the root of your query. Inside this folder, you will find all timestamps and per timestamp the path you requested.
+```
+your/path/waybackup_snapshots/
+└── the_root_of_your_query/ (example.com/)
+    ├── yyyymmddhhmmss/
+    │   ├── subidr1/
+    │   │   └── subdir2/
+    │   │       └── assets/
+    │   │           ├── image.jpg
+    │   │           └── style.css
+    ├── yyyymmddhhmmss/
+    │   ├── subdir1/
+    │   │   └── subdir2/
+    │   │       └── assets/
+    │   │           ├── image.jpg
+    │   │           └── style.css
+    ...
+```
+
+
+### Json Response
+
+For download queries:
+
+```
+[
+   {
+      "file": "/your/path/waybackup_snapshots/example.com/yyyymmddhhmmss/index.html",
+      "id": 1,
+      "redirect_timestamp": "yyyymmddhhmmss",
+      "redirect_url": "http://web.archive.org/web/yyyymmddhhmmssid_/http://example.com/",
+      "response": 200,
+      "timestamp": "yyyymmddhhmmss",
+      "url_archive": "http://web.archive.org/web/yyyymmddhhmmssid_/http://example.com/",
+      "url_origin": "http://example.com/"
+   },
+    ...
+]
+```
+
+For list queries:
+
+```
+[
+   {
+      "digest": "DIGESTOFSNAPSHOT",
+      "id": 1,
+      "mimetype": "text/html",
+      "status": "200",
+      "timestamp": "yyyymmddhhmmss",
+      "url": "http://example.com/"
+   },
+   ...
+]
+```
+
+## CSV Output
+
+The csv contains the json response in a table format.
+
+## Contributing
+
+I'm always happy for some feature requests to improve the usability of this script.
+Feel free to give suggestions and report issues. Project is still far from being perfect.

pywaybackup-0.8.1/PKG-INFO

@@ -1,104 +0,0 @@
-Metadata-Version: 2.1
-Name: pywaybackup
-Version: 0.8.1
-Summary: Download snapshots from the Wayback Machine
-Home-page: https://github.com/bitdruid/python-wayback-machine-downloader
-Author: bitdruid
-Author-email: bitdruid@outlook.com
-License: MIT
-Keywords: wayback machine internet archive
-Description-Content-Type: text/markdown
-License-File: LICENSE
-
-# archive wayback downloader
-
-[![PyPI](https://img.shields.io/pypi/v/pywaybackup)](https://pypi.org/project/pywaybackup/)
-[![PyPI - Downloads](https://img.shields.io/pypi/dm/pywaybackup)](https://pypi.org/project/pywaybackup/)
-![Release](https://img.shields.io/badge/Release-beta-orange)
-![Python Version](https://img.shields.io/badge/Python-3.6-blue)
-[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
-
-Downloading archived web pages from the [Wayback Machine](https://archive.org/web/).
-
-Internet-archive is a nice source for several OSINT-information. This script is a work in progress to query and fetch archived web pages.
-
-## Installation
-
-### Pip
-
-1. Install the package <br>
-   ```pip install pywaybackup```
-2. Run the script <br>
-   ```waybackup -h```
-
-### Manual
-
-1. Clone the repository <br>
-   ```git clone https://github.com/bitdruid/python-wayback-machine-downloader.git```
-2. Install <br>
-   ```pip install .```
-   - in a virtual env or use `--break-system-package`
-
-## Usage
-
-This script allows you to download content from the Wayback Machine (archive.org). You can use it to download either the latest version or all versions of web page snapshots within a specified range.
-
-<!-- ## Info -->
-
-### Arguments
-
-- `-h`, `--help`: Show the help message and exit.
-- `-a`, `--about`: Show information about the script and exit.
-
-#### Required Arguments
-
-- `-u URL`, `--url URL`: The URL of the web page to download. This argument is required.
-
-#### Mode Selection (Choose One)
-
-- `-c`, `--current`: Download the latest version of each file snapshot. You will get a rebuild of the current website with all available files.
-- `-f`, `--full`: Download snapshots of all timestamps. You will get a folder per timestamp with the files available at that time.
-- `-s`, `--save`: Save a page to the Wayback Machine. (beta)
-
-#### Optional Arguments
-
-- `-l`, `--list`: Only print the snapshots available within the specified range. Does not download the snapshots.
-- `-e`, `--explicit`: Only download the explicit given url. No wildcard subdomains or paths.
-- `-o OUTPUT`, `--output OUTPUT`: The folder where downloaded files will be saved.
-
-- **Range Selection:**<br>
-Specify the range in years or a specific timestamp either start, end or both. If you specify the `range` argument, the `start` and `end` arguments will be ignored. Format for timestamps: YYYYMMDDhhmmss. You can only give a year or increase specificity by going through the timestamp starting on the left.<br>
-(year 2019, year+month 201901, year+month+day 20190101, year+month+day+hour 2019010112)
-   - `-r RANGE`, `--range RANGE`: Specify the range in years for which to search and download snapshots.
-   - `--start`: Timestamp to start searching.
-   - `--end`: Timestamp to end searching.
-
-#### Additional
-
-- `--csv`: Save a csv file with the list of snapshots inside the output folder.
-- `--no-redirect`: Do not follow redirects of snapshots. Archive.org sometimes redirects to a different snapshot for several reasons. Downloading redirects may lead to timestamp-folders which contain some files with a different timestamp. This does not matter if you only want to download the latest version (`-c`).
-- `--verbosity [LEVEL]`: Set the verbosity: json (print json response), progress (show progress bar) or standard (default).
-- `--retry [RETRY_FAILED]`: Retry failed downloads. You can specify the number of retry attempts as an integer.
-- `--worker [AMOUNT]`: The number of worker to use for downloading (simultaneous downloads). Default is 1. A safe spot is about 10 workers. Beware: Using too many worker will lead into refused connections from the Wayback Machine. Duration about 1.5 minutes.
-
-### Examples
-
-Download latest snapshot of all files:<br>
-`waybackup -u http://example.com -c`
-
-Download latest snapshot of all files with retries:<br>
-`waybackup -u http://example.com -c --retry 3`
-
-Download all snapshots sorted per timestamp with a specified range and follow redirects:<br>
-`waybackup -u http://example.com -f -r 5 --redirect`
-
-Download all snapshots sorted per timestamp with a specified range and save to a specified folder with 3 worker:<br>
-`waybackup -u http://example.com -f -r 5 -o /home/user/Downloads/snapshots --worker 3`
-
-List available snapshots per timestamp without downloading:<br>
-`waybackup -u http://example.com -f -l`
-
-## Contributing
-
-I'm always happy for some feature requests to improve the usability of this script.
-Feel free to give suggestions and report issues. Project is still far from being perfect.

pywaybackup-0.8.1/README.md

@@ -1,92 +0,0 @@
-# archive wayback downloader
-
-[![PyPI](https://img.shields.io/pypi/v/pywaybackup)](https://pypi.org/project/pywaybackup/)
-[![PyPI - Downloads](https://img.shields.io/pypi/dm/pywaybackup)](https://pypi.org/project/pywaybackup/)
-![Release](https://img.shields.io/badge/Release-beta-orange)
-![Python Version](https://img.shields.io/badge/Python-3.6-blue)
-[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
-
-Downloading archived web pages from the [Wayback Machine](https://archive.org/web/).
-
-Internet-archive is a nice source for several OSINT-information. This script is a work in progress to query and fetch archived web pages.
-
-## Installation
-
-### Pip
-
-1. Install the package <br>
-   ```pip install pywaybackup```
-2. Run the script <br>
-   ```waybackup -h```
-
-### Manual
-
-1. Clone the repository <br>
-   ```git clone https://github.com/bitdruid/python-wayback-machine-downloader.git```
-2. Install <br>
-   ```pip install .```
-   - in a virtual env or use `--break-system-package`
-
-## Usage
-
-This script allows you to download content from the Wayback Machine (archive.org). You can use it to download either the latest version or all versions of web page snapshots within a specified range.
-
-<!-- ## Info -->
-
-### Arguments
-
-- `-h`, `--help`: Show the help message and exit.
-- `-a`, `--about`: Show information about the script and exit.
-
-#### Required Arguments
-
-- `-u URL`, `--url URL`: The URL of the web page to download. This argument is required.
-
-#### Mode Selection (Choose One)
-
-- `-c`, `--current`: Download the latest version of each file snapshot. You will get a rebuild of the current website with all available files.
-- `-f`, `--full`: Download snapshots of all timestamps. You will get a folder per timestamp with the files available at that time.
-- `-s`, `--save`: Save a page to the Wayback Machine. (beta)
-
-#### Optional Arguments
-
-- `-l`, `--list`: Only print the snapshots available within the specified range. Does not download the snapshots.
-- `-e`, `--explicit`: Only download the explicit given url. No wildcard subdomains or paths.
-- `-o OUTPUT`, `--output OUTPUT`: The folder where downloaded files will be saved.
-
-- **Range Selection:**<br>
-Specify the range in years or a specific timestamp either start, end or both. If you specify the `range` argument, the `start` and `end` arguments will be ignored. Format for timestamps: YYYYMMDDhhmmss. You can only give a year or increase specificity by going through the timestamp starting on the left.<br>
-(year 2019, year+month 201901, year+month+day 20190101, year+month+day+hour 2019010112)
-   - `-r RANGE`, `--range RANGE`: Specify the range in years for which to search and download snapshots.
-   - `--start`: Timestamp to start searching.
-   - `--end`: Timestamp to end searching.
-
-#### Additional
-
-- `--csv`: Save a csv file with the list of snapshots inside the output folder.
-- `--no-redirect`: Do not follow redirects of snapshots. Archive.org sometimes redirects to a different snapshot for several reasons. Downloading redirects may lead to timestamp-folders which contain some files with a different timestamp. This does not matter if you only want to download the latest version (`-c`).
-- `--verbosity [LEVEL]`: Set the verbosity: json (print json response), progress (show progress bar) or standard (default).
-- `--retry [RETRY_FAILED]`: Retry failed downloads. You can specify the number of retry attempts as an integer.
-- `--worker [AMOUNT]`: The number of worker to use for downloading (simultaneous downloads). Default is 1. A safe spot is about 10 workers. Beware: Using too many worker will lead into refused connections from the Wayback Machine. Duration about 1.5 minutes.
-
-### Examples
-
-Download latest snapshot of all files:<br>
-`waybackup -u http://example.com -c`
-
-Download latest snapshot of all files with retries:<br>
-`waybackup -u http://example.com -c --retry 3`
-
-Download all snapshots sorted per timestamp with a specified range and follow redirects:<br>
-`waybackup -u http://example.com -f -r 5 --redirect`
-
-Download all snapshots sorted per timestamp with a specified range and save to a specified folder with 3 worker:<br>
-`waybackup -u http://example.com -f -r 5 -o /home/user/Downloads/snapshots --worker 3`
-
-List available snapshots per timestamp without downloading:<br>
-`waybackup -u http://example.com -f -l`
-
-## Contributing
-
-I'm always happy for some feature requests to improve the usability of this script.
-Feel free to give suggestions and report issues. Project is still far from being perfect.

pywaybackup-0.8.1/pywaybackup/__version__.py

@@ -1 +0,0 @@
-__version__ = "0.8.1"

pywaybackup-0.8.1/pywaybackup.egg-info/PKG-INFO

@@ -1,104 +0,0 @@
-Metadata-Version: 2.1
-Name: pywaybackup
-Version: 0.8.1
-Summary: Download snapshots from the Wayback Machine
-Home-page: https://github.com/bitdruid/python-wayback-machine-downloader
-Author: bitdruid
-Author-email: bitdruid@outlook.com
-License: MIT
-Keywords: wayback machine internet archive
-Description-Content-Type: text/markdown
-License-File: LICENSE
-
-# archive wayback downloader
-
-[![PyPI](https://img.shields.io/pypi/v/pywaybackup)](https://pypi.org/project/pywaybackup/)
-[![PyPI - Downloads](https://img.shields.io/pypi/dm/pywaybackup)](https://pypi.org/project/pywaybackup/)
-![Release](https://img.shields.io/badge/Release-beta-orange)
-![Python Version](https://img.shields.io/badge/Python-3.6-blue)
-[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
-
-Downloading archived web pages from the [Wayback Machine](https://archive.org/web/).
-
-Internet-archive is a nice source for several OSINT-information. This script is a work in progress to query and fetch archived web pages.
-
-## Installation
-
-### Pip
-
-1. Install the package <br>
-   ```pip install pywaybackup```
-2. Run the script <br>
-   ```waybackup -h```
-
-### Manual
-
-1. Clone the repository <br>
-   ```git clone https://github.com/bitdruid/python-wayback-machine-downloader.git```
-2. Install <br>
-   ```pip install .```
-   - in a virtual env or use `--break-system-package`
-
-## Usage
-
-This script allows you to download content from the Wayback Machine (archive.org). You can use it to download either the latest version or all versions of web page snapshots within a specified range.
-
-<!-- ## Info -->
-
-### Arguments
-
-- `-h`, `--help`: Show the help message and exit.
-- `-a`, `--about`: Show information about the script and exit.
-
-#### Required Arguments
-
-- `-u URL`, `--url URL`: The URL of the web page to download. This argument is required.
-
-#### Mode Selection (Choose One)
-
-- `-c`, `--current`: Download the latest version of each file snapshot. You will get a rebuild of the current website with all available files.
-- `-f`, `--full`: Download snapshots of all timestamps. You will get a folder per timestamp with the files available at that time.
-- `-s`, `--save`: Save a page to the Wayback Machine. (beta)
-
-#### Optional Arguments
-
-- `-l`, `--list`: Only print the snapshots available within the specified range. Does not download the snapshots.
-- `-e`, `--explicit`: Only download the explicit given url. No wildcard subdomains or paths.
-- `-o OUTPUT`, `--output OUTPUT`: The folder where downloaded files will be saved.
-
-- **Range Selection:**<br>
-Specify the range in years or a specific timestamp either start, end or both. If you specify the `range` argument, the `start` and `end` arguments will be ignored. Format for timestamps: YYYYMMDDhhmmss. You can only give a year or increase specificity by going through the timestamp starting on the left.<br>
-(year 2019, year+month 201901, year+month+day 20190101, year+month+day+hour 2019010112)
-   - `-r RANGE`, `--range RANGE`: Specify the range in years for which to search and download snapshots.
-   - `--start`: Timestamp to start searching.
-   - `--end`: Timestamp to end searching.
-
-#### Additional
-
-- `--csv`: Save a csv file with the list of snapshots inside the output folder.
-- `--no-redirect`: Do not follow redirects of snapshots. Archive.org sometimes redirects to a different snapshot for several reasons. Downloading redirects may lead to timestamp-folders which contain some files with a different timestamp. This does not matter if you only want to download the latest version (`-c`).
-- `--verbosity [LEVEL]`: Set the verbosity: json (print json response), progress (show progress bar) or standard (default).
-- `--retry [RETRY_FAILED]`: Retry failed downloads. You can specify the number of retry attempts as an integer.
-- `--worker [AMOUNT]`: The number of worker to use for downloading (simultaneous downloads). Default is 1. A safe spot is about 10 workers. Beware: Using too many worker will lead into refused connections from the Wayback Machine. Duration about 1.5 minutes.
-
-### Examples
-
-Download latest snapshot of all files:<br>
-`waybackup -u http://example.com -c`
-
-Download latest snapshot of all files with retries:<br>
-`waybackup -u http://example.com -c --retry 3`
-
-Download all snapshots sorted per timestamp with a specified range and follow redirects:<br>
-`waybackup -u http://example.com -f -r 5 --redirect`
-
-Download all snapshots sorted per timestamp with a specified range and save to a specified folder with 3 worker:<br>
-`waybackup -u http://example.com -f -r 5 -o /home/user/Downloads/snapshots --worker 3`
-
-List available snapshots per timestamp without downloading:<br>
-`waybackup -u http://example.com -f -l`
-
-## Contributing
-
-I'm always happy for some feature requests to improve the usability of this script.
-Feel free to give suggestions and report issues. Project is still far from being perfect.