wayback2warc 1.0.0__tar.gz

wayback2warc-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 tmctmt
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

wayback2warc-1.0.0/PKG-INFO

@@ -0,0 +1,101 @@
+Metadata-Version: 2.4
+Name: wayback2warc
+Version: 1.0.0
+Summary: WARC exporter for the Wayback Machine
+Author-email: tmctmt <tmctmt@proton.me>
+License: MIT License
+        
+        Copyright (c) 2026 tmctmt
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+        
+Keywords: Internet Archive,Wayback Machine,WARC
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Environment :: Console
+Classifier: Intended Audience :: System Administrators
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Natural Language :: English
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python
+Classifier: Topic :: Utilities
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: httpx[socks]==0.28.1
+Requires-Dist: tqdm==4.67.3
+Requires-Dist: warcio==1.8.1
+Dynamic: license-file
+
+# wayback2warc
+A CLI tool that exports captures from the Wayback Machine and packs them into WARC files.
+
+```
+usage: wayback2warc [-h] [-c COLLAPSE] [-f FILTER] [-m] [-p PROXY] [-t THREADS] [-w WARC_SIZE] url [prefix]
+
+positional arguments:
+  url                   *.example.org or example.org/*
+  prefix                output path prefix
+
+options:
+  -h, --help            show this help message and exit
+  -c, --collapse COLLAPSE
+                        dedupe records based on returned key
+  -f, --filter FILTER   filter records based on returned boolean
+  -m, --meta            dump cdx metadata into jsonl file
+  -p, --proxy PROXY     url formatted proxy, can be a single value or a file
+  -t, --threads THREADS
+                        concurrent downloads (default: 2)
+  -w, --warc-size WARC_SIZE
+                        max size of produced warc files in MB (default: 1024)
+```
+
+## Install
+`pip install wayback2warc`
+
+## Usage
+
+Download all subdomains, all pages, and all captures for a domain:
+
+`wayback2warc '*.example.org'`
+
+Download all pages and all captures for a prefix:
+
+`wayback2warc 'example.org/*'`
+
+Download all captures for a single page:
+
+`wayback2warc 'example.org'`
+
+Download all pages but only a yearly capture of each:
+
+`wayback2warc 'example.org/*' --collapse 'lambda m: (m.urlkey, m.timestamp[:4])'`
+
+## Filtering and collapsing
+This tool implements filtering/collapsing functionality on the client-side rather than relying on the CDX server to do so.
+This is a necessity due to the way CDX paging works, but on the bright side it allows for much more granular queries.
+
+Instead of a query language, lambda functions are used to narrow down captures for downloading.
+A [CaptureMetadata](https://github.com/tmctmt/wayback2warc/blob/main/wayback2warc.py#L28) instance is passed to both functions, and the returned value is used as a key for collapsing, or as a boolean for filtering.
+
+Collapse captures by URL excluding query:
+`--collapse 'lambda m: m.urlkey.split("?")[0]'`
+
+Filter captures before 2020 and exclude images:
+`--filter 'lambda m: m.timestamp < "2020" and "image/" not in m.mimetype'`
+

wayback2warc-1.0.0/README.md

@@ -0,0 +1,57 @@
+# wayback2warc
+A CLI tool that exports captures from the Wayback Machine and packs them into WARC files.
+
+```
+usage: wayback2warc [-h] [-c COLLAPSE] [-f FILTER] [-m] [-p PROXY] [-t THREADS] [-w WARC_SIZE] url [prefix]
+
+positional arguments:
+  url                   *.example.org or example.org/*
+  prefix                output path prefix
+
+options:
+  -h, --help            show this help message and exit
+  -c, --collapse COLLAPSE
+                        dedupe records based on returned key
+  -f, --filter FILTER   filter records based on returned boolean
+  -m, --meta            dump cdx metadata into jsonl file
+  -p, --proxy PROXY     url formatted proxy, can be a single value or a file
+  -t, --threads THREADS
+                        concurrent downloads (default: 2)
+  -w, --warc-size WARC_SIZE
+                        max size of produced warc files in MB (default: 1024)
+```
+
+## Install
+`pip install wayback2warc`
+
+## Usage
+
+Download all subdomains, all pages, and all captures for a domain:
+
+`wayback2warc '*.example.org'`
+
+Download all pages and all captures for a prefix:
+
+`wayback2warc 'example.org/*'`
+
+Download all captures for a single page:
+
+`wayback2warc 'example.org'`
+
+Download all pages but only a yearly capture of each:
+
+`wayback2warc 'example.org/*' --collapse 'lambda m: (m.urlkey, m.timestamp[:4])'`
+
+## Filtering and collapsing
+This tool implements filtering/collapsing functionality on the client-side rather than relying on the CDX server to do so.
+This is a necessity due to the way CDX paging works, but on the bright side it allows for much more granular queries.
+
+Instead of a query language, lambda functions are used to narrow down captures for downloading.
+A [CaptureMetadata](https://github.com/tmctmt/wayback2warc/blob/main/wayback2warc.py#L28) instance is passed to both functions, and the returned value is used as a key for collapsing, or as a boolean for filtering.
+
+Collapse captures by URL excluding query:
+`--collapse 'lambda m: m.urlkey.split("?")[0]'`
+
+Filter captures before 2020 and exclude images:
+`--filter 'lambda m: m.timestamp < "2020" and "image/" not in m.mimetype'`
+

wayback2warc-1.0.0/pyproject.toml

@@ -0,0 +1,32 @@
+[build-system]
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "wayback2warc"
+version = "1.0.0"
+description = "WARC exporter for the Wayback Machine"
+readme = "README.md"
+authors = [
+    { name = "tmctmt", email = "tmctmt@proton.me" },
+]
+license = { file = "LICENSE" }
+requires-python = ">=3.8"
+classifiers = [
+    "Development Status :: 5 - Production/Stable",
+    "Environment :: Console",
+    "Intended Audience :: System Administrators",
+    "License :: OSI Approved :: MIT License",
+    "Natural Language :: English",
+    "Operating System :: OS Independent",
+    "Programming Language :: Python",
+    "Topic :: Utilities",
+]
+keywords = ["Internet Archive", "Wayback Machine", "WARC"]
+dependencies = [
+    "httpx[socks]==0.28.1",
+    "tqdm==4.67.3",
+    "warcio==1.8.1",
+]
+[project.scripts]
+wayback2warc = "wayback2warc.py"

wayback2warc-1.0.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

wayback2warc-1.0.0/wayback2warc.egg-info/PKG-INFO

@@ -0,0 +1,101 @@
+Metadata-Version: 2.4
+Name: wayback2warc
+Version: 1.0.0
+Summary: WARC exporter for the Wayback Machine
+Author-email: tmctmt <tmctmt@proton.me>
+License: MIT License
+        
+        Copyright (c) 2026 tmctmt
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+        
+Keywords: Internet Archive,Wayback Machine,WARC
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Environment :: Console
+Classifier: Intended Audience :: System Administrators
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Natural Language :: English
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python
+Classifier: Topic :: Utilities
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: httpx[socks]==0.28.1
+Requires-Dist: tqdm==4.67.3
+Requires-Dist: warcio==1.8.1
+Dynamic: license-file
+
+# wayback2warc
+A CLI tool that exports captures from the Wayback Machine and packs them into WARC files.
+
+```
+usage: wayback2warc [-h] [-c COLLAPSE] [-f FILTER] [-m] [-p PROXY] [-t THREADS] [-w WARC_SIZE] url [prefix]
+
+positional arguments:
+  url                   *.example.org or example.org/*
+  prefix                output path prefix
+
+options:
+  -h, --help            show this help message and exit
+  -c, --collapse COLLAPSE
+                        dedupe records based on returned key
+  -f, --filter FILTER   filter records based on returned boolean
+  -m, --meta            dump cdx metadata into jsonl file
+  -p, --proxy PROXY     url formatted proxy, can be a single value or a file
+  -t, --threads THREADS
+                        concurrent downloads (default: 2)
+  -w, --warc-size WARC_SIZE
+                        max size of produced warc files in MB (default: 1024)
+```
+
+## Install
+`pip install wayback2warc`
+
+## Usage
+
+Download all subdomains, all pages, and all captures for a domain:
+
+`wayback2warc '*.example.org'`
+
+Download all pages and all captures for a prefix:
+
+`wayback2warc 'example.org/*'`
+
+Download all captures for a single page:
+
+`wayback2warc 'example.org'`
+
+Download all pages but only a yearly capture of each:
+
+`wayback2warc 'example.org/*' --collapse 'lambda m: (m.urlkey, m.timestamp[:4])'`
+
+## Filtering and collapsing
+This tool implements filtering/collapsing functionality on the client-side rather than relying on the CDX server to do so.
+This is a necessity due to the way CDX paging works, but on the bright side it allows for much more granular queries.
+
+Instead of a query language, lambda functions are used to narrow down captures for downloading.
+A [CaptureMetadata](https://github.com/tmctmt/wayback2warc/blob/main/wayback2warc.py#L28) instance is passed to both functions, and the returned value is used as a key for collapsing, or as a boolean for filtering.
+
+Collapse captures by URL excluding query:
+`--collapse 'lambda m: m.urlkey.split("?")[0]'`
+
+Filter captures before 2020 and exclude images:
+`--filter 'lambda m: m.timestamp < "2020" and "image/" not in m.mimetype'`
+

wayback2warc-1.0.0/wayback2warc.egg-info/SOURCES.txt

@@ -0,0 +1,10 @@
+LICENSE
+README.md
+pyproject.toml
+wayback2warc.py
+wayback2warc.egg-info/PKG-INFO
+wayback2warc.egg-info/SOURCES.txt
+wayback2warc.egg-info/dependency_links.txt
+wayback2warc.egg-info/entry_points.txt
+wayback2warc.egg-info/requires.txt
+wayback2warc.egg-info/top_level.txt

wayback2warc-1.0.0/wayback2warc.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

wayback2warc-1.0.0/wayback2warc.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+wayback2warc = wayback2warc.py

wayback2warc-1.0.0/wayback2warc.egg-info/requires.txt

@@ -0,0 +1,3 @@
+httpx[socks]==0.28.1
+tqdm==4.67.3
+warcio==1.8.1

wayback2warc-1.0.0/wayback2warc.egg-info/top_level.txt

@@ -0,0 +1 @@
+wayback2warc

wayback2warc-1.0.0/wayback2warc.py

@@ -0,0 +1,227 @@
+from collections import deque
+from concurrent.futures import ThreadPoolExecutor, as_completed
+from dataclasses import dataclass, astuple
+from datetime import datetime, timezone
+from io import BytesIO
+from operator import attrgetter
+import argparse
+import json
+import os.path
+import random
+import re
+import sys
+import uuid
+
+from tqdm import tqdm
+from warcio import WARCWriter, StatusAndHeaders
+import httpx
+
+@dataclass(frozen=True)
+class Capture:
+    url: str
+    date: datetime
+    statusline: str
+    headers: list[tuple[str, str]]
+    content: bytes
+
+@dataclass(frozen=True)
+class CaptureMetadata:
+    urlkey: str
+    timestamp: str
+    original: str
+    mimetype: str
+    statuscode: str
+    digest: str
+    length: str
+
+class PooledClient:
+    base_url = 'https://web.archive.org'
+    timeout = httpx.Timeout(30, connect=5, read=30)
+    max_retries = 30
+
+    def __init__(self, proxy_list: list[str | None]):
+        self.proxy_list = proxy_list
+        self.session_pool = deque()
+
+    def request(self, method: str, url: str, retry: int = 1, **kwargs):
+        try:
+            session = self.session_pool.popleft()
+        except IndexError:
+            proxy = random.choice(self.proxy_list)
+            session = httpx.Client(base_url=self.base_url, timeout=self.timeout, proxy=proxy)
+
+        try:
+            resp = session.request(method, url, **kwargs)
+            # we're blocked if this header is not present
+            # exempt error 400 from retries as the cause is usually url encoding issues on our part
+            assert 'x-app-server' in resp.headers or resp.status_code == 400
+        except:
+            session.close()
+            if retry > self.max_retries:
+                raise
+            return self.request(method, url, retry+1, **kwargs)
+        else:
+            self.session_pool.append(session)
+            return resp
+        
+    def get_cdx_page_count(self, url: str):
+        resp = self.request('GET', '/cdx/search/cdx', params={
+            'url': url,
+            'showNumPages': True
+        })
+        return int(resp.text)
+
+    def get_cdx_page(self, url: str, page: int):
+        resp = self.request('GET', '/cdx/search/cdx', params={
+            'url': url,
+            'page': page,
+            'output': 'json'
+        })
+        rows = resp.json()
+        return [CaptureMetadata(*row) for row in rows[1:]]
+    
+    def get_capture(self, url: str, timestamp: str):
+        try:
+            resp = self.request('GET', f'/web/{timestamp}id_/{url}')
+        except:
+            # .request already handles retries
+            return
+
+        # capture is not available
+        if 'memento-datetime' not in resp.headers:
+            return
+        
+        headers = [('content-length', f'{len(resp.content)}')]
+        for header, value in resp.headers.multi_items():
+            if header.startswith('x-archive-orig-'):
+                header = header[15:]
+            elif header == 'location':
+                value = re.sub(r'^(https?://web\.archive\.org)?/web/\d{14}\w*/', '', value)
+            elif header == 'content-type':
+                pass
+            else:
+                continue
+            if header not in ('content-length', 'content-encoding', 'transfer-encoding'):
+                headers.append((header, value))
+
+        try:
+            original = resp.links['original']['url']
+        except KeyError:
+            original = url
+        date = datetime.strptime(timestamp, '%Y%m%d%H%M%S').replace(tzinfo=timezone.utc)
+        statusline = f'HTTP/1.1 {resp.status_code} {resp.reason_phrase}'
+
+        return Capture(original, date, statusline, headers, resp.content)
+
+def unordered_map(pool, fn, iterable):
+    return (future.result() for future in as_completed(pool.submit(fn, item) for item in iterable))
+
+if __name__ == '__main__':
+    arg_parser = argparse.ArgumentParser(
+        description='WARC exporter for the Wayback Machine',
+        epilog=(
+            "capture fields: urlkey, timestamp, original, mimetype, statuscode, digest, length\n"
+            "\n"
+            "examples:\n"
+            "%(prog)s '*.example.org' --collapse 'lambda m: m.digest'\n"
+            "%(prog)s 'twitter.com/elonmusk/status/*' -c 'lambda m: m.urlkey.split(\"?\")[0]' -f 'lambda m: m.timestamp < \"2022\"'"
+        ),
+        formatter_class=argparse.RawDescriptionHelpFormatter
+    )
+    arg_parser.add_argument('url',
+                            help='*.example.org or example.org/*')
+    arg_parser.add_argument('prefix', nargs='?', default='',
+                            help='output path prefix')
+    arg_parser.add_argument('-c', '--collapse', type=eval,
+                            help='dedupe records based on returned key')
+    arg_parser.add_argument('-f', '--filter', type=eval,
+                            help='filter records based on returned boolean')
+    arg_parser.add_argument('-m', '--meta', action='store_true',
+                            help='dump cdx metadata into jsonl file')
+    arg_parser.add_argument('-p', '--proxy',
+                            help='url formatted proxy, can be a single value or a file')
+    arg_parser.add_argument('-t', '--threads', type=int, default=2,
+                            help='concurrent downloads (default: %(default)s)')
+    arg_parser.add_argument('-w', '--warc-size', type=int, default=1024,
+                            help='max size of produced warc files in MB (default: %(default)s)')
+    args = arg_parser.parse_args()
+
+    if not args.proxy:
+        proxy_list = [None]
+    elif os.path.exists(args.proxy):
+        with open(args.proxy) as file:
+            proxy_list = file.read().splitlines()
+    else:
+        proxy_list = [args.proxy]
+
+    client = PooledClient(proxy_list)
+    pages = client.get_cdx_page_count(args.url)
+    pool = ThreadPoolExecutor(max_workers=args.threads)
+    queue: list[CaptureMetadata] = []
+
+    with tqdm(total=pages, desc='listing cdx') as progress:
+        keys = set()
+
+        for chunk in unordered_map(pool, lambda page: client.get_cdx_page(args.url, page), range(pages)):
+            for meta in chunk:
+                if int(meta.length) > 1024**2 * 100:
+                    print(f'skipping large file: web.archive.org/web/{meta.timestamp}/{meta.original}', file=sys.stderr)
+                    continue
+
+                if args.filter and not args.filter(meta):
+                    continue
+
+                if args.collapse:
+                    key = args.collapse(meta)
+                    if key in keys:
+                        continue
+                    keys.add(key)
+
+                queue.append(meta)
+
+            progress.set_postfix({'queued': len(queue)})
+            progress.update()
+            
+        del keys
+    
+    queue.sort(key=attrgetter('timestamp'))
+
+    if args.meta:
+        with open(f'{args.prefix}meta.jsonl', 'w') as file:
+            for meta in queue:
+                file.write(json.dumps(astuple(meta)) + '\n')
+        sys.exit()
+
+    with tqdm(total=len(queue), desc='downloading captures') as progress:
+        file = None
+        skipped = 0
+
+        for capture in unordered_map(pool, lambda meta: client.get_capture(meta.original, meta.timestamp), queue):
+            if not capture:
+                skipped += 1
+                progress.set_postfix({'skipped': skipped})
+                progress.update()
+                continue
+            
+            if not file or file.tell() > 1024**2 * args.warc_size:
+                if file:
+                    file.close()
+                file = open(f'{args.prefix}{uuid.uuid4()}.warc.gz', 'xb')
+                writer = WARCWriter(file, gzip=True)
+
+            record = writer.create_warc_record(
+                uri=capture.url,
+                record_type='response',
+                payload=BytesIO(capture.content),
+                warc_headers_dict={'WARC-Date': capture.date.isoformat().replace('+00:00', 'Z')},
+                http_headers=StatusAndHeaders(capture.statusline, capture.headers)
+            )
+            deterministic_uuid = uuid.uuid5(uuid.NAMESPACE_URL, f'{capture.url}|{capture.date.isoformat()}')
+            record.rec_headers.replace_header('WARC-Record-ID', f'<urn:uuid:{deterministic_uuid}>')
+            writer.write_record(record)
+            progress.update()
+
+        if file:
+            file.close()
+
+    pool.shutdown()