pywaybackup 1.0.1__tar.gz → 1.0.2__tar.gz

{pywaybackup-1.0.1 → pywaybackup-1.0.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: pywaybackup
-Version: 1.0.1
+Version: 1.0.2
 Summary: Download snapshots from the Wayback Machine
 Home-page: https://github.com/bitdruid/python-wayback-machine-downloader
 Author: bitdruid
@@ -9,6 +9,8 @@ License: MIT
 Keywords: wayback machine internet archive
 Description-Content-Type: text/markdown
 License-File: LICENSE
+Requires-Dist: requests==2.31.0
+Requires-Dist: tqdm==4.66.2
 
 # archive wayback downloader
 
@@ -60,7 +62,7 @@ This script allows you to download content from the Wayback Machine (archive.org
 #### Optional Arguments
 
 - `-l`, `--list`: Only print the snapshots available within the specified range. Does not download the snapshots.
-- `-e`, `--explicit`: Only download the explicit given url. No wildcard subdomains or paths.
+- `-e`, `--explicit`: Only download the explicit given url. No wildcard subdomains or paths. Use e.g. to get root-only snapshots.
 - `-o`, `--output`: The folder where downloaded files will be saved.
 
 - **Range Selection:**<br>
@@ -83,8 +85,8 @@ Specify the range in years or a specific timestamp either start, end or both. If
 Download latest snapshot of all files:<br>
 `waybackup -u http://example.com -c`
 
-Download latest snapshot of all files with retries:<br>
-`waybackup -u http://example.com -c --retry 3`
+Download latest snapshot of a specific file:<br>
+`waybackup -u http://example.com/subdir/file.html -c`
 
 Download all snapshots sorted per timestamp with a specified range and do not follow redirects:<br>
 `waybackup -u http://example.com -f -r 5 --no-redirect`
@@ -101,6 +103,84 @@ Download all snapshots and output a json response:<br>
 List available snapshots per timestamp without downloading and save a csv file to home folder:<br>
 `waybackup -u http://example.com -f -l --csv /home/user/Downloads`
 
+## Output path structure
+
+The output path is currently structured as follows by an example for the query:<br>
+`http://example.com/subdir1/subdir2/assets/`:
+<br><br>
+For the current version (`-c`):
+- The requested path will only include all files/folders starting from your query-path.
+```
+your/path/waybackup_snapshots/
+└── the_root_of_your_query/ (example.com/)
+    └── subdir1/
+        └── subdir2/
+            └── assets/
+                ├── image.jpg
+                ├── style.css
+                ...
+```
+For all versions (`-f`):
+- Will currently create a folder named as the root of your query. Inside this folder, you will find all timestamps and per timestamp the path you requested.
+```
+your/path/waybackup_snapshots/
+└── the_root_of_your_query/ (example.com/)
+    ├── yyyymmddhhmmss/
+    │   ├── subidr1/
+    │   │   └── subdir2/
+    │   │       └── assets/
+    │   │           ├── image.jpg
+    │   │           └── style.css
+    ├── yyyymmddhhmmss/
+    │   ├── subdir1/
+    │   │   └── subdir2/
+    │   │       └── assets/
+    │   │           ├── image.jpg
+    │   │           └── style.css
+    ...
+```
+
+
+### Json Response
+
+For download queries:
+
+```
+[
+   {
+      "file": "/your/path/waybackup_snapshots/example.com/yyyymmddhhmmss/index.html",
+      "id": 1,
+      "redirect_timestamp": "yyyymmddhhmmss",
+      "redirect_url": "http://web.archive.org/web/yyyymmddhhmmssid_/http://example.com/",
+      "response": 200,
+      "timestamp": "yyyymmddhhmmss",
+      "url_archive": "http://web.archive.org/web/yyyymmddhhmmssid_/http://example.com/",
+      "url_origin": "http://example.com/"
+   },
+    ...
+]
+```
+
+For list queries:
+
+```
+[
+   {
+      "digest": "DIGESTOFSNAPSHOT",
+      "id": 1,
+      "mimetype": "text/html",
+      "status": "200",
+      "timestamp": "yyyymmddhhmmss",
+      "url": "http://example.com/"
+   },
+   ...
+]
+```
+
+## CSV Output
+
+The csv contains the json response in a table format.
+
 ## Contributing
 
 I'm always happy for some feature requests to improve the usability of this script.

{pywaybackup-1.0.1 → pywaybackup-1.0.2}/README.md

@@ -48,7 +48,7 @@ This script allows you to download content from the Wayback Machine (archive.org
 #### Optional Arguments
 
 - `-l`, `--list`: Only print the snapshots available within the specified range. Does not download the snapshots.
-- `-e`, `--explicit`: Only download the explicit given url. No wildcard subdomains or paths.
+- `-e`, `--explicit`: Only download the explicit given url. No wildcard subdomains or paths. Use e.g. to get root-only snapshots.
 - `-o`, `--output`: The folder where downloaded files will be saved.
 
 - **Range Selection:**<br>
@@ -71,8 +71,8 @@ Specify the range in years or a specific timestamp either start, end or both. If
 Download latest snapshot of all files:<br>
 `waybackup -u http://example.com -c`
 
-Download latest snapshot of all files with retries:<br>
-`waybackup -u http://example.com -c --retry 3`
+Download latest snapshot of a specific file:<br>
+`waybackup -u http://example.com/subdir/file.html -c`
 
 Download all snapshots sorted per timestamp with a specified range and do not follow redirects:<br>
 `waybackup -u http://example.com -f -r 5 --no-redirect`
@@ -89,6 +89,84 @@ Download all snapshots and output a json response:<br>
 List available snapshots per timestamp without downloading and save a csv file to home folder:<br>
 `waybackup -u http://example.com -f -l --csv /home/user/Downloads`
 
+## Output path structure
+
+The output path is currently structured as follows by an example for the query:<br>
+`http://example.com/subdir1/subdir2/assets/`:
+<br><br>
+For the current version (`-c`):
+- The requested path will only include all files/folders starting from your query-path.
+```
+your/path/waybackup_snapshots/
+└── the_root_of_your_query/ (example.com/)
+    └── subdir1/
+        └── subdir2/
+            └── assets/
+                ├── image.jpg
+                ├── style.css
+                ...
+```
+For all versions (`-f`):
+- Will currently create a folder named as the root of your query. Inside this folder, you will find all timestamps and per timestamp the path you requested.
+```
+your/path/waybackup_snapshots/
+└── the_root_of_your_query/ (example.com/)
+    ├── yyyymmddhhmmss/
+    │   ├── subidr1/
+    │   │   └── subdir2/
+    │   │       └── assets/
+    │   │           ├── image.jpg
+    │   │           └── style.css
+    ├── yyyymmddhhmmss/
+    │   ├── subdir1/
+    │   │   └── subdir2/
+    │   │       └── assets/
+    │   │           ├── image.jpg
+    │   │           └── style.css
+    ...
+```
+
+
+### Json Response
+
+For download queries:
+
+```
+[
+   {
+      "file": "/your/path/waybackup_snapshots/example.com/yyyymmddhhmmss/index.html",
+      "id": 1,
+      "redirect_timestamp": "yyyymmddhhmmss",
+      "redirect_url": "http://web.archive.org/web/yyyymmddhhmmssid_/http://example.com/",
+      "response": 200,
+      "timestamp": "yyyymmddhhmmss",
+      "url_archive": "http://web.archive.org/web/yyyymmddhhmmssid_/http://example.com/",
+      "url_origin": "http://example.com/"
+   },
+    ...
+]
+```
+
+For list queries:
+
+```
+[
+   {
+      "digest": "DIGESTOFSNAPSHOT",
+      "id": 1,
+      "mimetype": "text/html",
+      "status": "200",
+      "timestamp": "yyyymmddhhmmss",
+      "url": "http://example.com/"
+   },
+   ...
+]
+```
+
+## CSV Output
+
+The csv contains the json response in a table format.
+
 ## Contributing
 
 I'm always happy for some feature requests to improve the usability of this script.

{pywaybackup-1.0.1 → pywaybackup-1.0.2}/pywaybackup/SnapshotCollection.py

@@ -36,11 +36,11 @@ class SnapshotCollection:
     @classmethod
     def create_collection(cls):
         new_collection = []
-        for cdx_entry in cls.SNAPSHOT_COLLECTION:
+        for idx, cdx_entry in enumerate(cls.SNAPSHOT_COLLECTION):
             timestamp, url = cdx_entry["timestamp"], cdx_entry["url"]
             url_archive = f"http://web.archive.org/web/{timestamp}{cls._url_get_filetype(url)}/{url}"
             collection_entry = {
-                "id": cls.SNAPSHOT_COLLECTION.index(cdx_entry),
+                "id": idx,
                 "timestamp": timestamp,
                 "url_archive": url_archive,
                 "url_origin": url,
@@ -65,7 +65,7 @@ class SnapshotCollection:
         - download_file: The output path for the snapshot entry (str) with filename.
         """
         timestamp, url = collection_entry["timestamp"], collection_entry["url_origin"]
-        domain, subdir, filename = cls._url_split(url)
+        domain, subdir, filename = cls.url_split(url, index=True)
         if cls.MODE_CURRENT:
             download_dir = os.path.join(output, domain, subdir)
         else:
@@ -109,12 +109,17 @@ class SnapshotCollection:
         return urltype
 
     @classmethod
-    def _url_split(cls, url):
+    def url_split(cls, url, index=False):
         """
         Split a URL into domain, subdir and filename.
         """
+        if not urlparse(url).scheme:
+            url = "http://" + url
         parsed_url = urlparse(url)
-        domain = parsed_url.netloc
-        subdir = parsed_url.path.strip("/").rsplit("/", 1)[0]
-        filename = parsed_url.path.split("/")[-1] or "index.html"
+        domain = parsed_url.netloc.split("@")[-1].split(":")[0] # split mailto: and port
+        filename = parsed_url.path.split("/")[-1]
+        if index is True and filename == "":
+            filename = "index.html"
+        subdir = parsed_url.path.strip("/").replace(filename, "").strip("/")
+        filename = filename.replace("%20", " ") # replace url encoded spaces
         return domain, subdir, filename

pywaybackup-1.0.2/pywaybackup/__version__.py

@@ -0,0 +1 @@
+__version__ = "1.0.2"

{pywaybackup-1.0.1 → pywaybackup-1.0.2}/pywaybackup/archive.py

@@ -89,12 +89,25 @@ def query_list(url: str, range: int, start: int, end: int, explicit: bool, mode:
     try:
         v.write("\nQuerying snapshots...")
         query_range = ""
+        
         if not range:
             if start: query_range = query_range + f"&from={start}"
             if end: query_range = query_range + f"&to={end}"
         else: 
             query_range = "&from=" + str(datetime.now().year - range)
-        cdx_url = f"*.{url}/*" if not explicit else f"{url}"
+
+        # parse user input url and create according cdx url
+        domain, subdir, filename = sc.url_split(url)
+        if domain and not subdir and not filename:
+            cdx_url = f"*.{domain}/*" if not explicit else f"{domain}"
+        if domain and subdir and not filename:
+            cdx_url = f"{domain}/{subdir}/*"
+        if domain and subdir and filename:
+            cdx_url = f"{domain}/{subdir}/{filename}/*"
+        if domain and not subdir and filename:
+            cdx_url = f"{domain}/{filename}/*"
+
+        print(f"---> {cdx_url}")
         cdxQuery = f"https://web.archive.org/cdx/search/xd?output=json&url={cdx_url}{query_range}&fl=timestamp,digest,mimetype,statuscode,original&filter!=statuscode:200"
         cdxResult = requests.get(cdxQuery)
         sc.create_list(cdxResult, mode)
@@ -121,6 +134,7 @@ def download_list(output, retry, no_redirect, workers):
     else:
         batch_size = sc.count_list()
     sc.create_collection()
+    v.write("\n-----> Snapshots prepared")
     batch_list = [sc.SNAPSHOT_COLLECTION[i:i + batch_size] for i in range(0, len(sc.SNAPSHOT_COLLECTION), batch_size)]    
     threads = []
     worker = 0

{pywaybackup-1.0.1 → pywaybackup-1.0.2}/pywaybackup.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: pywaybackup
-Version: 1.0.1
+Version: 1.0.2
 Summary: Download snapshots from the Wayback Machine
 Home-page: https://github.com/bitdruid/python-wayback-machine-downloader
 Author: bitdruid
@@ -9,6 +9,8 @@ License: MIT
 Keywords: wayback machine internet archive
 Description-Content-Type: text/markdown
 License-File: LICENSE
+Requires-Dist: requests==2.31.0
+Requires-Dist: tqdm==4.66.2
 
 # archive wayback downloader
 
@@ -60,7 +62,7 @@ This script allows you to download content from the Wayback Machine (archive.org
 #### Optional Arguments
 
 - `-l`, `--list`: Only print the snapshots available within the specified range. Does not download the snapshots.
-- `-e`, `--explicit`: Only download the explicit given url. No wildcard subdomains or paths.
+- `-e`, `--explicit`: Only download the explicit given url. No wildcard subdomains or paths. Use e.g. to get root-only snapshots.
 - `-o`, `--output`: The folder where downloaded files will be saved.
 
 - **Range Selection:**<br>
@@ -83,8 +85,8 @@ Specify the range in years or a specific timestamp either start, end or both. If
 Download latest snapshot of all files:<br>
 `waybackup -u http://example.com -c`
 
-Download latest snapshot of all files with retries:<br>
-`waybackup -u http://example.com -c --retry 3`
+Download latest snapshot of a specific file:<br>
+`waybackup -u http://example.com/subdir/file.html -c`
 
 Download all snapshots sorted per timestamp with a specified range and do not follow redirects:<br>
 `waybackup -u http://example.com -f -r 5 --no-redirect`
@@ -101,6 +103,84 @@ Download all snapshots and output a json response:<br>
 List available snapshots per timestamp without downloading and save a csv file to home folder:<br>
 `waybackup -u http://example.com -f -l --csv /home/user/Downloads`
 
+## Output path structure
+
+The output path is currently structured as follows by an example for the query:<br>
+`http://example.com/subdir1/subdir2/assets/`:
+<br><br>
+For the current version (`-c`):
+- The requested path will only include all files/folders starting from your query-path.
+```
+your/path/waybackup_snapshots/
+└── the_root_of_your_query/ (example.com/)
+    └── subdir1/
+        └── subdir2/
+            └── assets/
+                ├── image.jpg
+                ├── style.css
+                ...
+```
+For all versions (`-f`):
+- Will currently create a folder named as the root of your query. Inside this folder, you will find all timestamps and per timestamp the path you requested.
+```
+your/path/waybackup_snapshots/
+└── the_root_of_your_query/ (example.com/)
+    ├── yyyymmddhhmmss/
+    │   ├── subidr1/
+    │   │   └── subdir2/
+    │   │       └── assets/
+    │   │           ├── image.jpg
+    │   │           └── style.css
+    ├── yyyymmddhhmmss/
+    │   ├── subdir1/
+    │   │   └── subdir2/
+    │   │       └── assets/
+    │   │           ├── image.jpg
+    │   │           └── style.css
+    ...
+```
+
+
+### Json Response
+
+For download queries:
+
+```
+[
+   {
+      "file": "/your/path/waybackup_snapshots/example.com/yyyymmddhhmmss/index.html",
+      "id": 1,
+      "redirect_timestamp": "yyyymmddhhmmss",
+      "redirect_url": "http://web.archive.org/web/yyyymmddhhmmssid_/http://example.com/",
+      "response": 200,
+      "timestamp": "yyyymmddhhmmss",
+      "url_archive": "http://web.archive.org/web/yyyymmddhhmmssid_/http://example.com/",
+      "url_origin": "http://example.com/"
+   },
+    ...
+]
+```
+
+For list queries:
+
+```
+[
+   {
+      "digest": "DIGESTOFSNAPSHOT",
+      "id": 1,
+      "mimetype": "text/html",
+      "status": "200",
+      "timestamp": "yyyymmddhhmmss",
+      "url": "http://example.com/"
+   },
+   ...
+]
+```
+
+## CSV Output
+
+The csv contains the json response in a table format.
+
 ## Contributing
 
 I'm always happy for some feature requests to improve the usability of this script.

pywaybackup-1.0.1/pywaybackup/__version__.py

@@ -1 +0,0 @@
-__version__ = "1.0.1"