pywget 2.0.2__tar.gz → 2.0.4__tar.gz

{pywget-2.0.2/pywget.egg-info → pywget-2.0.4}/PKG-INFO

@@ -1,370 +1,611 @@
-Metadata-Version: 2.4
-Name: pywget
-Version: 2.0.2
-Summary: Robust, resumable, multi-connection wget alternative - CLI and Python library, zero dependencies
-Author-email: Hadi Cahyadi <cumulus13@gmail.com>
-Maintainer-email: cumulus13 <cumulus13@gmail.com>
-License: BSD 3-Clause License
-        
-        Copyright (c) 2021-2026, Hadi Cahyadi
-        All rights reserved.
-        
-        Redistribution and use in source and binary forms, with or without
-        modification, are permitted provided that the following conditions are met:
-        
-        1. Redistributions of source code must retain the above copyright notice, this
-           list of conditions and the following disclaimer.
-        
-        2. Redistributions in binary form must reproduce the above copyright notice,
-           this list of conditions and the following disclaimer in the documentation
-           and/or other materials provided with the distribution.
-        
-        3. Neither the name of the copyright holder nor the names of its
-           contributors may be used to endorse or promote products derived from
-           this software without specific prior written permission.
-        
-        THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
-        AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
-        IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
-        DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
-        FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
-        DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
-        SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
-        CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
-        OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-        OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-        
-Project-URL: Homepage, https://github.com/cumulus13/pywget
-Project-URL: Documentation, https://github.com/cumulus13/pywget#readme
-Project-URL: Source, https://github.com/cumulus13/pywget
-Project-URL: Issues, https://github.com/cumulus13/pywget/issues
-Project-URL: Changelog, https://github.com/cumulus13/pywget/blob/master/CHANGELOG.md
-Keywords: wget,download,downloader,resume,multi-connection,cli,http
-Classifier: Development Status :: 5 - Production/Stable
-Classifier: Environment :: Console
-Classifier: Intended Audience :: Developers
-Classifier: Intended Audience :: System Administrators
-Classifier: License :: OSI Approved :: BSD License
-Classifier: Operating System :: OS Independent
-Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.8
-Classifier: Programming Language :: Python :: 3.9
-Classifier: Programming Language :: Python :: 3.10
-Classifier: Programming Language :: Python :: 3.11
-Classifier: Programming Language :: Python :: 3.12
-Classifier: Programming Language :: Python :: 3.13
-Classifier: Topic :: Internet :: WWW/HTTP
-Classifier: Topic :: System :: Networking
-Classifier: Typing :: Typed
-Requires-Python: >=3.8
-Description-Content-Type: text/markdown
-License-File: LICENSE
-Provides-Extra: dev
-Requires-Dist: pytest>=7; extra == "dev"
-Requires-Dist: pytest-cov; extra == "dev"
-Requires-Dist: ruff; extra == "dev"
-Requires-Dist: mypy; extra == "dev"
-Requires-Dist: build; extra == "dev"
-Requires-Dist: twine; extra == "dev"
-Dynamic: license-file
-
-# pywget
-
-[![CI](https://github.com/cumulus13/pywget/actions/workflows/ci.yml/badge.svg)](https://github.com/cumulus13/pywget/actions/workflows/ci.yml)
-[![PyPI](https://img.shields.io/pypi/v/pywget.svg)](https://pypi.org/project/pywget/)
-[![Python](https://img.shields.io/pypi/pyversions/pywget.svg)](https://pypi.org/project/pywget/)
-[![License](https://img.shields.io/badge/license-BSD-blue.svg)](LICENSE)
-
-A robust, resumable, **multi-connection** command-line downloader and Python library.
-A drop-in-style `wget` alternative with zero third-party dependencies.
-
-## Features
-
-| Area | What you get |
-|------|--------------|
-| Speed | Parallel segmented download (`-n 1..32`), automatic single-stream fallback |
-| Reliability | Crash-safe resume, retries with exponential backoff + jitter, `Retry-After`, atomic rename |
-| Integrity | `--checksum sha256:...`, size verification, disk-space pre-check |
-| Security | http/https only, credentials stripped on cross-origin redirects, path-traversal-proof filenames, TLS verification on by default |
-| Control | Bandwidth limit, custom headers, basic auth, proxy, custom CA, timeouts |
-| Automation | Batch input (`-i`), `--json` output, stable exit codes, SIGINT/SIGTERM-safe |
-| Library | Typed API, progress callbacks, cancel support, legacy `wget.download()` kept |
-| Portability | Python 3.8+, Windows / Linux / macOS, no dependencies |
-
-## Installation
-
-```bash
-pip install pywget
-```
-
-> **v2 note:** Python 2 is no longer supported. Python 3.8+ is required. The CLI and
-> `wget.download(url, dir)` keep working as before.
-
-## Quick start
-
-```bash
-# simple download into a directory (same as before)
-pywget "https://example.com/file.iso" -p ~/Downloads
-
-# 8 parallel connections, 2 MiB/s cap, verify checksum
-pywget https://example.com/file.iso -n 8 --limit-rate 2M --checksum sha256:<hex>
-
-# many URLs from a file (or stdin with -i -), JSON report
-pywget -i urls.txt -p downloads --json
-
-# auth + custom header
-pywget https://api.example.com/export.zip --user me -H "Accept: application/zip"
-```
-
-Interrupted (Ctrl+C, crash, network loss)? Run the same command again - it resumes.
-
-## CLI reference
-
-| Option | Description |
-|--------|-------------|
-| `URL ...` | One or more http(s) URLs |
-| `-i, --input-file FILE` | Read URLs from file (`-` = stdin); `#` comments allowed |
-| `-p, -P, --path DIR` | Download directory (default `.`) |
-| `-O, --output NAME` | Output file name (single URL) |
-| `-n, --connections N` | Parallel connections per file, 1-32 (default 4) |
-| `-r, --retries N` | Retries per request/segment (default 5) |
-| `-T, --timeout SEC` | Network timeout (default 30) |
-| `--limit-rate RATE` | Total speed cap, e.g. `500k`, `2M` |
-| `-H, --header 'K: V'` | Extra header (repeatable) |
-| `-U, --user-agent UA` | Custom User-Agent |
-| `--user`, `--password` | Basic auth (password also from `PYWGET_PASSWORD` or prompt) |
-| `--proxy URL` | Proxy (default: environment) |
-| `--no-check-certificate` | Disable TLS verification (not recommended) |
-| `--ca-cert FILE` | Custom CA bundle |
-| `--checksum ALGO:HEX` | Verify result (md5, sha1, sha256, sha512, ...) |
-| `--overwrite` | Replace existing file and restart |
-| `--no-resume` | Never resume partial files |
-| `--no-mtime` | Do not copy `Last-Modified` to the file |
-| `-q`, `-v` | Quiet / debug logging |
-| `--json` | Machine-readable results on stdout |
-| `-c` | Accepted for wget compatibility (resume is default) |
-
-### Exit codes
-
-| Code | Meaning |
-|------|---------|
-| 0 | Success |
-| 1 | Unexpected error |
-| 2 | Bad usage / invalid URL |
-| 3 | Network error |
-| 4 | HTTP error status (404, 403, ...) |
-| 5 | Checksum mismatch |
-| 6 | Storage / I/O error (disk full, permissions) |
-| 130 | Interrupted (partial data kept for resume) |
-
-## Python API
-
-```python
-import pywget
-
-path = pywget.download(
-    "https://example.com/file.iso",
-    "/data/downloads",
-    progress=True,          # terminal bar
-    connections=8,
-    limit_rate="5M",
-    checksum="sha256:ab12...",
-)
-print(path)
-```
-
-Full control:
-
-```python
-from pywget import Downloader, Options, ProgressInfo
-
-def on_progress(p: ProgressInfo) -> None:
-    print(f"{p.filename}: {p.percent or 0:.1f}%  {p.speed/1e6:.1f} MB/s  ETA {p.eta}")
-
-dl = Downloader(Options(output_dir="out", retries=8, progress=on_progress,
-                        headers={"Authorization": "Bearer TOKEN"}))
-result = dl.fetch("https://example.com/big.bin")
-print(result.path, result.size, result.resumed, result.connections)
-
-# from another thread: dl.cancel()  -> partial state is kept for resume
-```
-
-Legacy API (unchanged):
-
-```python
-from pywget import wget
-
-wget.download("http://www.test.com/test.tar.gz", "/home/root/Downloads")
-```
-
-Errors derive from `pywget.PywgetError` (`InvalidURLError`, `NetworkError`,
-`HTTPStatusError`, `ChecksumError`, `StorageError`, `DownloadCancelled`).
-A `Downloader` runs one download at a time - use one instance per thread.
-
-## How it works
-
-### Architecture
-
-```mermaid
-flowchart LR
-    CLI["CLI<br/>pywget.cli"] --> DL
-    API["Python API<br/>pywget.download()"] --> DL
-    LEG["Legacy shim<br/>pywget.wget"] --> DL
-
-    subgraph Engine["pywget.core"]
-        DL["Downloader"] --> PROBE["Probe<br/>Range: bytes=0-0"]
-        PROBE --> PLAN{"Size known and<br/>ranges supported?"}
-        PLAN -- yes --> SEG["Segment workers<br/>1..N threads"]
-        PLAN -- no --> STR["Single stream<br/>worker"]
-        SEG --> MON["Monitor loop<br/>progress + state save"]
-        STR --> MON
-        MON --> VER["Verify<br/>size + checksum"]
-    end
-
-    SEG -. throttled by .-> RL["RateLimiter"]
-    STR -. throttled by .-> RL
-    MON --> TR["Tracker / ProgressBar"]
-    SEG <--> ST[("state file<br/>.pywget")]
-    VER --> OUT[("final file<br/>atomic rename")]
-```
-
-[![Architecture 1](https://i.ibb.co/mr6hxjYg/pywget-architecture1.jpg)](https://i.ibb.co/mr6hxjYg/pywget-architecture1.jpg)
-
-### Download sequence
-
-```mermaid
-sequenceDiagram
-    autonumber
-    actor U as User
-    participant D as Downloader
-    participant S as Server
-    participant F as Disk
-
-    U->>D: fetch(url)
-    D->>S: GET Range: bytes=0-0
-    S-->>D: 206 + Content-Range (size, ETag)
-    D->>F: load .pywget state (if any)
-    alt state matches size / ETag / mtime
-        D->>D: resume remaining segments
-    else fresh start
-        D->>F: preallocate .part, split into N segments
-    end
-    par segment 1..N
-        D->>S: GET Range: bytes=a-b
-        S-->>D: 206 data
-        D->>F: write at offset
-    end
-    loop every second
-        D->>F: save state
-        D-->>U: progress callback
-    end
-    D->>F: verify size and checksum
-    D->>F: rename .part to final name
-    D-->>U: DownloadResult
-```
-
-[![Architecture 2](https://i.ibb.co/DPgLTD6D/pywget-architecture2.jpg)](https://i.ibb.co/DPgLTD6D/pywget-architecture2.jpg)
-
-### Segment lifecycle and retries
-
-```mermaid
-stateDiagram-v2
-    [*] --> Probing
-    Probing --> Planning: size and ranges known
-    Probing --> Streaming: no range support
-    Planning --> Downloading
-    Streaming --> Verifying
-    Downloading --> Retrying: timeout / reset / 5xx / 429
-    Retrying --> Downloading: backoff + jitter, resume at offset
-    Retrying --> Failed: retries exhausted
-    Downloading --> Interrupted: Ctrl+C / SIGTERM / cancel()
-    Interrupted --> Planning: re-run, state validated
-    Downloading --> Verifying: all segments complete
-    Verifying --> Completed: size and checksum OK
-    Verifying --> Failed: mismatch, partial files removed
-    Completed --> [*]
-    Failed --> [*]
-```
-
-[![Architecture 3](https://i.ibb.co/YBZKFsxG/pywget-architecture3.jpg)](https://i.ibb.co/YBZKFsxG/pywget-architecture3.jpg)
-
-### Resume decision
-
-```mermaid
-flowchart TD
-    A["Start"] --> B{".pywget state<br/>and .part exist?"}
-    B -- no --> N["Fresh download"]
-    B -- yes --> C{"URL, size, ETag,<br/>Last-Modified equal?"}
-    C -- no --> N
-    C -- yes --> D{"Segments valid and<br/>.part size correct?"}
-    D -- no --> N
-    D -- yes --> R["Resume each segment<br/>from its saved offset"]
-```
-
-[![Architecture 4](https://i.ibb.co/nqGmzdpw/pywget-architecture4.jpg)](https://i.ibb.co/nqGmzdpw/pywget-architecture4.jpg)
-
-### Main types
-
-```mermaid
-classDiagram
-    class Downloader {
-        +fetch(url) DownloadResult
-        +download(url) Path
-        +cancel()
-    }
-    class Options {
-        connections
-        retries
-        limit_rate
-        checksum
-        progress
-    }
-    class DownloadResult {
-        path
-        size
-        resumed
-        connections
-        checksum
-    }
-    class Tracker
-    class ProgressBar
-    class RateLimiter
-    Downloader o-- Options
-    Downloader --> DownloadResult
-    Downloader --> Tracker
-    Downloader --> RateLimiter
-    ProgressBar ..> Tracker : renders snapshots
-```
-
-[![Architecture 5](https://i.ibb.co/9HYQ3WrT/pywget-architecture5.jpg)](https://i.ibb.co/9HYQ3WrT/pywget-architecture5.jpg)
-
-## Behaviour details
-
-- **Probing:** one `Range: bytes=0-0` request yields size, range support, ETag, `Last-Modified` and `Content-Disposition`.
-- **Segments:** a file is split into `min(connections, size / 2 MiB)` parts; small files use one connection.
-- **Existing file:** same size means skipped; different size is saved as `name.1` (wget style) unless `--overwrite`.
-- **Partial files:** stored as `name.part` + `name.pywget`; removed on success. Changed remote file (size/ETag/mtime) restarts cleanly.
-- **Redirects:** only http/https; `Authorization`/`Cookie` are dropped when the origin changes.
-- **Not supported (by design):** FTP, recursive mirroring.
-
-## Development
-
-```bash
-git clone https://github.com/cumulus13/pywget && cd pywget
-pip install -e ".[dev]"
-pytest --cov=pywget
-ruff check pywget tests && mypy pywget
-python -m build
-```
-
-## License
-
-[BSD](LICENSE)
-
-## Author
-
-[Hadi Cahyadi](https://github.com/cumulus13)
-
-[![Buy Me a Coffee](https://www.buymeacoffee.com/assets/img/custom_images/orange_img.png)](https://www.buymeacoffee.com/cumulus13)
-[![Donate via Ko-fi](https://ko-fi.com/img/githubbutton_sm.svg)](https://ko-fi.com/cumulus13)
-[Support me on Patreon](https://www.patreon.com/cumulus13)
+Metadata-Version: 2.4
+Name: pywget
+Version: 2.0.4
+Summary: Robust, resumable, multi-connection wget alternative - CLI and Python library, zero dependencies
+Author-email: Hadi Cahyadi <cumulus13@gmail.com>
+Maintainer-email: cumulus13 <cumulus13@gmail.com>
+License: BSD 3-Clause License
+        
+        Copyright (c) 2021-2026, Hadi Cahyadi
+        All rights reserved.
+        
+        Redistribution and use in source and binary forms, with or without
+        modification, are permitted provided that the following conditions are met:
+        
+        1. Redistributions of source code must retain the above copyright notice, this
+           list of conditions and the following disclaimer.
+        
+        2. Redistributions in binary form must reproduce the above copyright notice,
+           this list of conditions and the following disclaimer in the documentation
+           and/or other materials provided with the distribution.
+        
+        3. Neither the name of the copyright holder nor the names of its
+           contributors may be used to endorse or promote products derived from
+           this software without specific prior written permission.
+        
+        THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+        AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+        IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+        DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+        FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+        DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+        SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+        CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+        OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+        OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+        
+Project-URL: Homepage, https://github.com/cumulus13/pywget
+Project-URL: Documentation, https://github.com/cumulus13/pywget#readme
+Project-URL: Source, https://github.com/cumulus13/pywget
+Project-URL: Issues, https://github.com/cumulus13/pywget/issues
+Project-URL: Changelog, https://github.com/cumulus13/pywget/blob/master/CHANGELOG.md
+Keywords: wget,download,downloader,resume,multi-connection,cli,http
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: System Administrators
+Classifier: License :: OSI Approved :: BSD License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Internet :: WWW/HTTP
+Classifier: Topic :: System :: Networking
+Classifier: Typing :: Typed
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == "dev"
+Requires-Dist: pytest-cov; extra == "dev"
+Requires-Dist: ruff; extra == "dev"
+Requires-Dist: mypy; extra == "dev"
+Requires-Dist: build; extra == "dev"
+Requires-Dist: twine; extra == "dev"
+Dynamic: license-file
+
+# pywget
+
+[![CI](https://github.com/cumulus13/pywget/actions/workflows/ci.yml/badge.svg)](https://github.com/cumulus13/pywget/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/pywget.svg)](https://pypi.org/project/pywget/)
+[![Python](https://img.shields.io/pypi/pyversions/pywget.svg)](https://pypi.org/project/pywget/)
+[![License](https://img.shields.io/badge/license-BSD-blue.svg)](LICENSE)
+
+A robust, resumable, **multi-connection** command-line downloader and Python library.
+A drop-in-style `wget` alternative with zero third-party dependencies.
+
+## Features
+
+| Area | What you get |
+|------|--------------|
+| Speed | Parallel segmented download (`-n 1..32`), automatic single-stream fallback |
+| Reliability | Crash-safe resume, retries with exponential backoff + jitter, `Retry-After`, atomic rename |
+| Integrity | `--checksum sha256:...`, size verification, disk-space pre-check |
+| Security | http/https only, credentials stripped on cross-origin redirects, path-traversal-proof filenames, TLS verification on by default |
+| Control | Bandwidth limit, custom headers, basic auth, proxy, custom CA, timeouts |
+| Automation | Batch input (`-i`), `--json` output, stable exit codes, SIGINT/SIGTERM-safe |
+| Server mode | `pywget serve`: persistent queue, HTTP/JSON API, SSE events, web UI, token auth |
+| Agent mode | `pywget agent` / `--server`: send links from anywhere; offline-tolerant relay to a server |
+| Library | Typed API, progress callbacks, cancel support, legacy `wget.download()` kept |
+| Portability | Python 3.8+, Windows / Linux / macOS, no dependencies |
+
+## Installation
+
+```bash
+pip install pywget
+```
+
+> **v2 note:** Python 2 is no longer supported. Python 3.8+ is required. The CLI and
+> `wget.download(url, dir)` keep working as before.
+
+## Quick start
+
+```bash
+# simple download into a directory (same as before)
+pywget "https://example.com/file.iso" -p ~/Downloads
+
+# 8 parallel connections, 2 MiB/s cap, verify checksum
+pywget https://example.com/file.iso -n 8 --limit-rate 2M --checksum sha256:<hex>
+
+# many URLs from a file (or stdin with -i -), JSON report
+pywget -i urls.txt -p downloads --json
+
+# auth + custom header
+pywget https://api.example.com/export.zip --user me -H "Accept: application/zip"
+```
+
+```bash
+# download manager mode: run a server, push links to it (see "Server mode and agent")
+pywget serve --root ~/Downloads/pywget &
+pywget send https://example.com/file.iso -r https://example.com/ --wait
+pywget jobs list
+```
+
+Interrupted (Ctrl+C, crash, network loss)? Run the same command again - it resumes.
+
+## CLI reference
+
+| Option | Description |
+|--------|-------------|
+| `URL ...` | http(s) URLs, `c` = read links from the clipboard, or a text file of links |
+| `-i, --input-file FILE` | Read URLs from file (`-` = stdin); `#` comments allowed |
+| `-p, -P, --path DIR` | Download directory (default `.`) |
+| `-O, --output NAME` | Output file name (single URL) |
+| `-n, --connections N` | Parallel connections per file, 1-32 (default 4) |
+| `-t, --retries N` | Retries per request/segment (default 5) |
+| `-r, --referrer URL` | Referer header (many sites require it) |
+| `-C, --cookie STR` | Cookie string `a=b; c=d` or JSON object |
+| `-T, --timeout SEC` | Network timeout (default 30) |
+| `--limit-rate RATE` | Total speed cap, e.g. `500k`, `2M` |
+| `-H, --header 'K: V'` | Extra header (repeatable) |
+| `-U, -ua, --user-agent UA` | Custom User-Agent |
+| `--user`, `--password` | Basic auth (password also from `PYWGET_PASSWORD` or prompt) |
+| `--proxy URL` | Proxy (default: environment) |
+| `--no-check-certificate` | Disable TLS verification (not recommended) |
+| `--ca-cert FILE` | Custom CA bundle |
+| `--checksum ALGO:HEX` | Verify result (md5, sha1, sha256, sha512, ...) |
+| `--overwrite` | Replace existing file and restart |
+| `--no-resume` | Never resume partial files |
+| `--no-mtime` | Do not copy `Last-Modified` to the file |
+| `-q`, `-v` | Quiet / debug logging |
+| `--json` | Machine-readable results on stdout |
+| `-c` | Accepted for wget compatibility (resume is default) |
+| `-s, --server URL` | Send the links to a pywget server/agent instead of downloading locally |
+| `--token TOKEN` | API token for `--server` (default: `$PYWGET_TOKEN` or token file) |
+| `-a, --add-only` | With `--server`: queue paused, do not start (like IDM's *add*) |
+| `--wait` | With `--server`: wait for the jobs to finish |
+
+### Exit codes
+
+| Code | Meaning |
+|------|---------|
+| 0 | Success |
+| 1 | Unexpected error |
+| 2 | Bad usage / invalid URL |
+| 3 | Network error |
+| 4 | HTTP error status (404, 403, ...) |
+| 5 | Checksum mismatch |
+| 6 | Storage / I/O error (disk full, permissions) |
+| 7 | Server / agent error (unreachable, 401, 409, ...) |
+| 130 | Interrupted (partial data kept for resume) |
+
+## Python API
+
+```python
+import pywget
+
+path = pywget.download(
+    "https://example.com/file.iso",
+    "/data/downloads",
+    progress=True,          # terminal bar
+    connections=8,
+    limit_rate="5M",
+    checksum="sha256:ab12...",
+)
+print(path)
+```
+
+Full control:
+
+```python
+from pywget import Downloader, Options, ProgressInfo
+
+def on_progress(p: ProgressInfo) -> None:
+    print(f"{p.filename}: {p.percent or 0:.1f}%  {p.speed/1e6:.1f} MB/s  ETA {p.eta}")
+
+dl = Downloader(Options(output_dir="out", retries=8, progress=on_progress,
+                        headers={"Authorization": "Bearer TOKEN"}))
+result = dl.fetch("https://example.com/big.bin")
+print(result.path, result.size, result.resumed, result.connections)
+
+# from another thread: dl.cancel()  -> partial state is kept for resume
+```
+
+Talk to a server / agent, or embed one:
+
+```python
+from pywget import ServerClient, ServerConfig, ServerHandle
+
+client = ServerClient("http://127.0.0.1:8765")      # token: $PYWGET_TOKEN or token file
+job = client.submit({"url": "https://example.com/f.iso", "path": "isos"})[0]
+print(client.wait(job["id"])["status"])
+
+cfg = ServerConfig(port=0, token="secret-token", root="/srv/dl", state_dir="/var/lib/pywget")
+with ServerHandle(cfg) as server:                    # background server (tests, embedding)
+    print(server.url)
+```
+
+Legacy API (unchanged):
+
+```python
+from pywget import wget
+
+wget.download("http://www.test.com/test.tar.gz", "/home/root/Downloads")
+```
+
+Errors derive from `pywget.PywgetError` (`InvalidURLError`, `NetworkError`,
+`HTTPStatusError`, `ChecksumError`, `StorageError`, `DownloadCancelled`).
+A `Downloader` runs one download at a time - use one instance per thread.
+
+## Server mode and agent
+
+Run pywget as a long-lived download manager and push links to it from anywhere - the
+same idea as handing a link to IDM, but over a small authenticated HTTP API.
+
+```bash
+# 1. start the server (jobs may only write below --root)
+pywget serve --root ~/Downloads/pywget            # http://127.0.0.1:8765, token auto-created
+
+# 2. send links (token is read from the token file on the same machine)
+pywget send https://example.com/a.iso -p isos -r https://example.com/ --wait
+pywget https://example.com/b.zip --server http://127.0.0.1:8765      # same, legacy form
+pywget send c                      # links found in the clipboard
+pywget send links.txt -a           # a text file of links, queued paused (add only)
+
+# 3. manage
+pywget jobs list
+pywget jobs pause|resume|cancel|retry|remove <id>
+```
+
+Open `http://127.0.0.1:8765/` for the web UI (paste the token once).
+
+### Agent (relay)
+
+An **agent** is a tiny server that does not download: it accepts links on a local port
+and forwards them to an upstream server, keeping them in a persistent outbox when the
+upstream is offline and mirroring progress back. Put it on the machine where links
+appear (laptop, browser helper script) and let a server/NAS do the downloading.
+
+```bash
+# on the download box
+pywget serve --host 0.0.0.0 --root /srv/downloads --tls-cert c.pem --tls-key k.pem
+
+# on your workstation
+pywget agent --upstream https://nas:8765 --upstream-token <TOKEN> --port 8766
+pywget send https://example.com/file.iso --server http://127.0.0.1:8766 --wait
+```
+
+### System overview
+
+```mermaid
+flowchart LR
+    CLI["pywget send / --server"] --> AG
+    BR["Browser helper,<br/>scripts, curl"] --> AG
+    subgraph WS["Workstation"]
+        AG["Agent<br/>pywget agent<br/>persistent outbox"]
+    end
+    AG -- "HTTPS + token<br/>forward job, poll status" --> API
+    CLI -. "direct" .-> API
+    subgraph SV["Download server"]
+        API["HTTP API + web UI<br/>auth, Host check, root jail"] --> MGR["JobManager<br/>queue, N workers"]
+        MGR --> ENG["Downloader engine<br/>segments, resume, retry"]
+        MGR <--> ST[("jobs.json<br/>0600")]
+        ENG --> FS[("download root")]
+    end
+```
+
+[![Architecture 7](https://i.ibb.co/96D8hz7/pywget-architecture7.jpg)](https://i.ibb.co/96D8hz7/pywget-architecture7.jpg)
+
+
+### Job flow
+
+```mermaid
+sequenceDiagram
+    autonumber
+    actor U as User
+    participant A as Agent
+    participant S as Server
+    participant W as Worker
+    U->>A: pywget send URL --wait
+    A->>A: persist job (outbox)
+    loop until accepted
+        A->>S: POST /api/v1/jobs (Bearer token)
+        S-->>A: 201 job id or error / offline then backoff
+    end
+    S->>W: enqueue
+    W->>W: Downloader.fetch (segments, resume)
+    loop every second
+        A->>S: GET /api/v1/jobs/id
+        S-->>A: status, percent, speed, ETA
+    end
+    W-->>S: completed (atomic rename in root)
+    A-->>U: OK path
+```
+
+[![Architecture 8](https://i.ibb.co/4n0GBmSD/pywget-architecture8.jpg)](https://i.ibb.co/4n0GBmSD/pywget-architecture8.jpg)
+
+### Job states
+
+```mermaid
+stateDiagram-v2
+    [*] --> queued: add
+    [*] --> paused: add --add-only
+    queued --> running: worker free
+    running --> paused: pause (partial data kept)
+    paused --> queued: resume
+    running --> completed
+    running --> failed: error after retries
+    running --> cancelled: cancel (partial files removed)
+    queued --> cancelled
+    paused --> cancelled
+    failed --> queued: retry
+    cancelled --> queued: retry
+    running --> queued: server shutdown (auto-resumes on start)
+    completed --> [*]: remove
+    failed --> [*]: remove
+    cancelled --> [*]: remove
+```
+
+[![Architecture 9](https://i.ibb.co/wNkVB4fp/pywget-architecture9.jpg)](https://i.ibb.co/wNkVB4fp/pywget-architecture9.jpg)
+
+### HTTP API
+
+All `/api/v1/*` calls need `Authorization: Bearer <token>` (or `X-Pywget-Token`).
+
+| Method and path | Purpose |
+|-----------------|---------|
+| `GET /health` | Liveness, version, mode (no auth) |
+| `GET /` | Web UI (no auth, token entered in the page) |
+| `POST /api/v1/jobs` | Create job(s): a job object, `{"urls": [...], ...common}` or `{"jobs": [...]}` |
+| `GET /api/v1/jobs[?status=]` | List jobs |
+| `GET /api/v1/jobs/{id}` | One job (secrets are never returned) |
+| `POST /api/v1/jobs/{id}/pause\|resume\|cancel\|retry` | Control a job |
+| `DELETE /api/v1/jobs/{id}` | Remove a finished job record |
+| `GET /api/v1/stats` | Counts, total speed |
+| `GET /api/v1/events` | Server-sent events (`event: job`) |
+
+Job fields: `url`, `filename`, `path` (folder below the root), `referrer`, `cookie`
+(string or object), `headers`, `user_agent`, `username`, `password`, `connections`,
+`checksum`, `limit_rate`, `overwrite`, `add_only`.
+
+```bash
+curl -s -H "Authorization: Bearer $(cat ~/.config/pywget/token)" \
+     -d '{"url":"https://example.com/f.iso","path":"isos","referrer":"https://example.com/"}' \
+     http://127.0.0.1:8765/api/v1/jobs
+```
+
+### Security model
+
+- **Token required** for every API call; generated once (`token` file, mode 0600) or given via `--token` / `$PYWGET_TOKEN`; compared in constant time.
+- **Loopback by default.** Binding elsewhere prints a warning; use `--tls-cert/--tls-key` or an HTTPS reverse proxy.
+- **Root jail:** job `path` can never leave `--root` (absolute paths and `..` are rejected, symlinks resolved).
+- **No CORS** and a `Host` header allow-list (`--allowed-host`) against DNS-rebinding and drive-by browser requests.
+- **`--deny-private-networks`** refuses URLs resolving to private/loopback/link-local addresses (SSRF guard for exposed servers).
+- Request bodies capped at 1 MiB; headers checked for CR/LF injection; credentials and cookies are stored in `jobs.json` (0600) and never returned by the API.
+- Not supported: POST-body downloads (IDM `post data`), FTP.
+
+## How it works
+
+### Download engine
+
+```mermaid
+flowchart LR
+    CLI["CLI<br/>pywget.cli"] --> DL
+    API["Python API<br/>pywget.download()"] --> DL
+    LEG["Legacy shim<br/>pywget.wget"] --> DL
+    JOB["LocalExecutor<br/>(server jobs)"] --> DL
+
+    subgraph Engine["pywget.core"]
+        DL["Downloader"] --> PROBE["Probe<br/>Range: bytes=0-0"]
+        PROBE --> PLAN{"Size known and<br/>ranges supported?"}
+        PLAN -- yes --> SEG["Segment workers<br/>1..N threads"]
+        PLAN -- no --> STR["Single stream<br/>worker"]
+        SEG --> MON["Monitor loop<br/>progress + state save"]
+        STR --> MON
+        MON --> VER["Verify<br/>size + checksum"]
+    end
+
+    SEG -. throttled by .-> RL["RateLimiter"]
+    STR -. throttled by .-> RL
+    MON --> TR["Tracker / ProgressBar"]
+    SEG <--> ST[("state file<br/>.pywget")]
+    VER --> OUT[("final file<br/>atomic rename")]
+```
+
+[![Architecture 1](https://i.ibb.co/BVCnttHQ/pywget-architecture1.jpg)](https://i.ibb.co/BVCnttHQ/pywget-architecture1.jpg)
+
+### Download sequence
+
+```mermaid
+sequenceDiagram
+    autonumber
+    actor U as User
+    participant D as Downloader
+    participant S as Server
+    participant F as Disk
+
+    U->>D: fetch(url)
+    D->>S: GET Range: bytes=0-0
+    S-->>D: 206 + Content-Range (size, ETag)
+    D->>F: load .pywget state (if any)
+    alt state matches size / ETag / mtime
+        D->>D: resume remaining segments
+    else fresh start
+        D->>F: preallocate .part, split into N segments
+    end
+    par segment 1..N
+        D->>S: GET Range: bytes=a-b
+        S-->>D: 206 data
+        D->>F: write at offset
+    end
+    loop every second
+        D->>F: save state
+        D-->>U: progress callback
+    end
+    D->>F: verify size and checksum
+    D->>F: rename .part to final name
+    D-->>U: DownloadResult
+```
+
+[![Architecture 2](https://i.ibb.co/DPgLTD6D/pywget-architecture2.jpg)](https://i.ibb.co/DPgLTD6D/pywget-architecture2.jpg)
+
+### Segment lifecycle and retries
+
+```mermaid
+stateDiagram-v2
+    [*] --> Probing
+    Probing --> Planning: size and ranges known
+    Probing --> Streaming: no range support
+    Planning --> Downloading
+    Streaming --> Verifying
+    Downloading --> Retrying: timeout / reset / 5xx / 429
+    Retrying --> Downloading: backoff + jitter, resume at offset
+    Retrying --> Failed: retries exhausted
+    Downloading --> Interrupted: Ctrl+C / SIGTERM / cancel()
+    Interrupted --> Planning: re-run, state validated
+    Downloading --> Verifying: all segments complete
+    Verifying --> Completed: size and checksum OK
+    Verifying --> Failed: mismatch, partial files removed
+    Completed --> [*]
+    Failed --> [*]
+```
+
+[![Architecture 3](https://i.ibb.co/YBZKFsxG/pywget-architecture3.jpg)](https://i.ibb.co/YBZKFsxG/pywget-architecture3.jpg)
+
+### Resume decision
+
+```mermaid
+flowchart TD
+    A["Start"] --> B{".pywget state<br/>and .part exist?"}
+    B -- no --> N["Fresh download"]
+    B -- yes --> C{"URL, size, ETag,<br/>Last-Modified equal?"}
+    C -- no --> N
+    C -- yes --> D{"Segments valid and<br/>.part size correct?"}
+    D -- no --> N
+    D -- yes --> R["Resume each segment<br/>from its saved offset"]
+```
+
+[![Architecture 4](https://i.ibb.co/nqGmzdpw/pywget-architecture4.jpg)](https://i.ibb.co/nqGmzdpw/pywget-architecture4.jpg)
+
+### Main types - engine
+
+```mermaid
+classDiagram
+    class Downloader {
+        +fetch(url) DownloadResult
+        +download(url) Path
+        +cancel()
+        +partial
+    }
+    class Options {
+        connections
+        retries
+        limit_rate
+        rate_limiter
+        checksum
+        progress
+    }
+    class DownloadResult {
+        path
+        size
+        resumed
+        connections
+        checksum
+    }
+    class Tracker
+    class ProgressBar
+    class RateLimiter
+    Downloader o-- Options
+    Downloader --> DownloadResult
+    Downloader --> Tracker
+    Downloader --> RateLimiter
+    ProgressBar ..> Tracker : renders snapshots
+```
+
+[![Architecture 5](https://i.ibb.co/d0kCy05W/pywget-architecture5.jpg)](https://i.ibb.co/d0kCy05W/pywget-architecture5.jpg)
+
+### Main types - server and agent
+
+```mermaid
+classDiagram
+    class ServerHandle {
+        +start()
+        +close()
+        +url
+    }
+    class PywgetHTTPServer
+    class ApiHandler {
+        auth, Host check
+        routes /api/v1/*
+    }
+    class JobManager {
+        +add(spec) Job
+        +pause(id)
+        +resume(id)
+        +cancel(id)
+        +retry(id)
+        +remove(id)
+    }
+    class Job
+    class JobStore
+    class EventBus
+    class LocalExecutor
+    class RelayExecutor
+    class ServerClient
+    class ServerConfig
+    ServerHandle --> PywgetHTTPServer
+    ServerHandle --> JobManager
+    PywgetHTTPServer --> ApiHandler
+    ApiHandler --> JobManager
+    JobManager o-- Job
+    JobManager --> JobStore : persists
+    JobManager --> EventBus : SSE
+    JobManager --> LocalExecutor : server mode
+    JobManager --> RelayExecutor : agent mode
+    LocalExecutor --> Downloader
+    RelayExecutor --> ServerClient : forwards to upstream
+    ServerHandle o-- ServerConfig
+```
+
+[![Architecture 5](https://i.ibb.co/zVMXCFtz/pywget-architecture6.jpg)](https://i.ibb.co/zVMXCFtz/pywget-architecture6.jpg)
+
+### Module map
+
+| Module | Role |
+|--------|------|
+| `pywget.core` | Download engine (`Downloader`, `Options`) |
+| `pywget.progress`, `pywget.utils`, `pywget.errors` | Progress bar, helpers, exceptions and exit codes |
+| `pywget.cli` | CLI: direct download, `serve`, `agent`, `send`, `jobs` |
+| `pywget.server` | HTTP API, SSE, web UI hosting (`ServerHandle`, `run_server`) |
+| `pywget.jobs` | `Job`, `JobManager`, persistence, `LocalExecutor` / `RelayExecutor` |
+| `pywget.client` | `ServerClient` (used by `send`, `jobs` and the agent) |
+| `pywget.config` | Token file, config dir, `ServerConfig` |
+| `pywget.clipboard` | `c` input: clipboard and URL extraction |
+| `pywget.webui` | Embedded single-page web UI |
+| `pywget.wget` | Backward-compatible `wget.download()` |
+
+## Behaviour details
+
+- **Probing:** one `Range: bytes=0-0` request yields size, range support, ETag, `Last-Modified` and `Content-Disposition`.
+- **Segments:** a file is split into `min(connections, size / 2 MiB)` parts; small files use one connection.
+- **Existing file:** same size means skipped; different size is saved as `name.1` (wget style) unless `--overwrite`.
+- **Partial files:** stored as `name.part` + `name.pywget`; removed on success. Changed remote file (size/ETag/mtime) restarts cleanly.
+- **Redirects:** only http/https; `Authorization`/`Cookie` are dropped when the origin changes.
+- **Server jobs:** persisted in `jobs.json`; a job running at shutdown is re-queued and resumes from its `.part` data.
+- **Not supported (by design):** FTP, recursive mirroring, POST-body downloads.
+
+## Development
+
+```bash
+git clone https://github.com/cumulus13/pywget && cd pywget
+pip install -e ".[dev]"
+pytest --cov=pywget
+ruff check pywget tests && mypy pywget
+python -m build
+```
+
+## License
+
+[BSD](LICENSE)
+
+## Author
+
+[Hadi Cahyadi](https://github.com/cumulus13)
+
+[![Buy Me a Coffee](https://www.buymeacoffee.com/assets/img/custom_images/orange_img.png)](https://www.buymeacoffee.com/cumulus13)
+[![Donate via Ko-fi](https://ko-fi.com/img/githubbutton_sm.svg)](https://ko-fi.com/cumulus13)
+[Support me on Patreon](https://www.patreon.com/cumulus13)