pywget 2.0.4__tar.gz → 2.0.6__tar.gz

{pywget-2.0.4/pywget.egg-info → pywget-2.0.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pywget
-Version: 2.0.4
+Version: 2.0.6
 Summary: Robust, resumable, multi-connection wget alternative - CLI and Python library, zero dependencies
 Author-email: Hadi Cahyadi <cumulus13@gmail.com>
 Maintainer-email: cumulus13 <cumulus13@gmail.com>
@@ -231,6 +231,58 @@ Errors derive from `pywget.PywgetError` (`InvalidURLError`, `NetworkError`,
 `HTTPStatusError`, `ChecksumError`, `StorageError`, `DownloadCancelled`).
 A `Downloader` runs one download at a time - use one instance per thread.
 
+## Configuration and API token
+
+### Config file (optional)
+
+There is no config file unless you create one; every option also has a command-line flag.
+
+```bash
+pywget config path     # where it lives
+pywget config init     # write a fully commented template there (mode 0600)
+pywget config show
+```
+
+| OS | Default location |
+|----|------------------|
+| Windows | `%APPDATA%\pywget\config.ini` |
+| Linux / macOS | `~/.config/pywget/config.ini` (`$XDG_CONFIG_HOME` respected) |
+
+Override the directory with `PYWGET_CONFIG_DIR` or the file with `PYWGET_CONFIG`.
+Precedence: **command line > environment > config file > built-in defaults**.
+Sections: `[download]`, `[client]`, `[serve]`, `[agent]` (the generated template lists every key).
+Unknown keys or bad values are reported with the file path and exit code 2.
+
+```ini
+[serve]
+root = C:\DOWNLOADS
+max_jobs = 3
+connections = 2
+
+[client]
+server = http://127.0.0.1:8765
+```
+
+Job queue state lives in `<config dir>/server/jobs.json` (`agent/` for an agent); change it with `state_dir`.
+
+### API token
+
+The server and agent **always** require a token.
+
+- **Default:** on the first `pywget serve` / `pywget agent` a random 256-bit token is generated and saved to `<config dir>/token` (mode 0600). It is reused on every start, and local `send` / `jobs` commands read it automatically.
+- **Show / create / set:**
+
+```bash
+pywget token show            # print the current token and where it comes from
+pywget token new             # generate and save a new one (restart the server afterwards)
+pywget token set "my-own-secret-123"   # or: echo -n ... | pywget token set -
+pywget token path
+```
+
+- **Override order:** `--token` > `$PYWGET_TOKEN` > token file.
+- **Format:** not arbitrary text - 16 to 256 characters from `A-Z a-z 0-9 . _ ~ + / = -` (safe inside an HTTP header). Weak or malformed tokens are rejected at startup.
+- **Remote clients:** copy the token to the other machine (`pywget token set -` there, or `export PYWGET_TOKEN=...`).
+
 ## Server mode and agent
 
 Run pywget as a long-lived download manager and push links to it from anywhere - the
@@ -573,7 +625,7 @@ classDiagram
 | `pywget.server` | HTTP API, SSE, web UI hosting (`ServerHandle`, `run_server`) |
 | `pywget.jobs` | `Job`, `JobManager`, persistence, `LocalExecutor` / `RelayExecutor` |
 | `pywget.client` | `ServerClient` (used by `send`, `jobs` and the agent) |
-| `pywget.config` | Token file, config dir, `ServerConfig` |
+| `pywget.config` | Config file, token file and validation, `ServerConfig` |
 | `pywget.clipboard` | `c` input: clipboard and URL extraction |
 | `pywget.webui` | Embedded single-page web UI |
 | `pywget.wget` | Backward-compatible `wget.download()` |
@@ -582,6 +634,8 @@ classDiagram
 
 - **Probing:** one `Range: bytes=0-0` request yields size, range support, ETag, `Last-Modified` and `Content-Disposition`.
 - **Segments:** a file is split into `min(connections, size / 2 MiB)` parts; small files use one connection.
+- **Rate-limited servers:** on HTTP 429 the number of simultaneous segment requests is reduced automatically (down to 1) instead of failing the job; failed jobs keep partial data, so `pywget jobs retry ID` resumes.
+- **Output hygiene:** messages and job lists show `host/../file` only - query strings (often signed tokens) are never printed.
 - **Existing file:** same size means skipped; different size is saved as `name.1` (wget style) unless `--overwrite`.
 - **Partial files:** stored as `name.part` + `name.pywget`; removed on success. Changed remote file (size/ETag/mtime) restarts cleanly.
 - **Redirects:** only http/https; `Authorization`/`Cookie` are dropped when the origin changes.

{pywget-2.0.4 → pywget-2.0.6}/README.md

@@ -161,6 +161,58 @@ Errors derive from `pywget.PywgetError` (`InvalidURLError`, `NetworkError`,
 `HTTPStatusError`, `ChecksumError`, `StorageError`, `DownloadCancelled`).
 A `Downloader` runs one download at a time - use one instance per thread.
 
+## Configuration and API token
+
+### Config file (optional)
+
+There is no config file unless you create one; every option also has a command-line flag.
+
+```bash
+pywget config path     # where it lives
+pywget config init     # write a fully commented template there (mode 0600)
+pywget config show
+```
+
+| OS | Default location |
+|----|------------------|
+| Windows | `%APPDATA%\pywget\config.ini` |
+| Linux / macOS | `~/.config/pywget/config.ini` (`$XDG_CONFIG_HOME` respected) |
+
+Override the directory with `PYWGET_CONFIG_DIR` or the file with `PYWGET_CONFIG`.
+Precedence: **command line > environment > config file > built-in defaults**.
+Sections: `[download]`, `[client]`, `[serve]`, `[agent]` (the generated template lists every key).
+Unknown keys or bad values are reported with the file path and exit code 2.
+
+```ini
+[serve]
+root = C:\DOWNLOADS
+max_jobs = 3
+connections = 2
+
+[client]
+server = http://127.0.0.1:8765
+```
+
+Job queue state lives in `<config dir>/server/jobs.json` (`agent/` for an agent); change it with `state_dir`.
+
+### API token
+
+The server and agent **always** require a token.
+
+- **Default:** on the first `pywget serve` / `pywget agent` a random 256-bit token is generated and saved to `<config dir>/token` (mode 0600). It is reused on every start, and local `send` / `jobs` commands read it automatically.
+- **Show / create / set:**
+
+```bash
+pywget token show            # print the current token and where it comes from
+pywget token new             # generate and save a new one (restart the server afterwards)
+pywget token set "my-own-secret-123"   # or: echo -n ... | pywget token set -
+pywget token path
+```
+
+- **Override order:** `--token` > `$PYWGET_TOKEN` > token file.
+- **Format:** not arbitrary text - 16 to 256 characters from `A-Z a-z 0-9 . _ ~ + / = -` (safe inside an HTTP header). Weak or malformed tokens are rejected at startup.
+- **Remote clients:** copy the token to the other machine (`pywget token set -` there, or `export PYWGET_TOKEN=...`).
+
 ## Server mode and agent
 
 Run pywget as a long-lived download manager and push links to it from anywhere - the
@@ -503,7 +555,7 @@ classDiagram
 | `pywget.server` | HTTP API, SSE, web UI hosting (`ServerHandle`, `run_server`) |
 | `pywget.jobs` | `Job`, `JobManager`, persistence, `LocalExecutor` / `RelayExecutor` |
 | `pywget.client` | `ServerClient` (used by `send`, `jobs` and the agent) |
-| `pywget.config` | Token file, config dir, `ServerConfig` |
+| `pywget.config` | Config file, token file and validation, `ServerConfig` |
 | `pywget.clipboard` | `c` input: clipboard and URL extraction |
 | `pywget.webui` | Embedded single-page web UI |
 | `pywget.wget` | Backward-compatible `wget.download()` |
@@ -512,6 +564,8 @@ classDiagram
 
 - **Probing:** one `Range: bytes=0-0` request yields size, range support, ETag, `Last-Modified` and `Content-Disposition`.
 - **Segments:** a file is split into `min(connections, size / 2 MiB)` parts; small files use one connection.
+- **Rate-limited servers:** on HTTP 429 the number of simultaneous segment requests is reduced automatically (down to 1) instead of failing the job; failed jobs keep partial data, so `pywget jobs retry ID` resumes.
+- **Output hygiene:** messages and job lists show `host/../file` only - query strings (often signed tokens) are never printed.
 - **Existing file:** same size means skipped; different size is saved as `name.1` (wget style) unless `--overwrite`.
 - **Partial files:** stored as `name.part` + `name.pywget`; removed on success. Changed remote file (size/ETag/mtime) restarts cleanly.
 - **Redirects:** only http/https; `Authorization`/`Cookie` are dropped when the origin changes.

{pywget-2.0.4 → pywget-2.0.6}/pyproject.toml

@@ -6,7 +6,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "pywget"
-version = "2.0.4"
+version = "2.0.6"
 description = "Robust, resumable, multi-connection wget alternative - CLI and Python library, zero dependencies"
 readme = "README.md"
 requires-python = ">=3.8"
@@ -78,6 +78,9 @@ pywget = [
     "py.typed",
 ]
 
+[tool.setuptools.dynamic.version]
+attr = "pywget.__version__.__version__"
+
 [tool.pytest.ini_options]
 testpaths = [
     "tests",

{pywget-2.0.4 → pywget-2.0.6}/pywget/cli.py

@@ -15,10 +15,11 @@ import getpass
 import json
 import logging
 import os
+import shutil
 import signal
 import sys
 from pathlib import Path
-from typing import Any, Dict, List, Optional
+from typing import Any, Callable, Dict, List, Optional, Tuple
 
 from .__version__ import __version__
 from .client import ServerClient
@@ -29,10 +30,17 @@ from .config import (
     DEFAULT_PORT,
     ServerConfig,
     config_dir,
-    load_token,
+    config_file_path,
+    generate_token,
+    load_config,
+    resolve_token,
+    save_token,
+    token_path,
+    validate_token,
+    write_default_config,
 )
 from .core import Downloader, Options
-from .errors import PywgetError
+from .errors import ConfigError, PywgetError
 from .progress import ProgressBar
 from .utils import build_headers, format_size
 
@@ -49,12 +57,14 @@ other commands (pywget COMMAND -h):
   agent   run a relay that forwards links to an upstream server
   send    send links to a server/agent ("c" = clipboard, or a text file of links)
   jobs    list / show / pause / resume / cancel / retry / remove server jobs
+  token   show / new / set / path   - the API token (auto-generated on first serve)
+  config  path / init / show        - the optional config file (config.ini)
 
 exit codes: 0 ok | 2 bad usage/URL | 3 network | 4 HTTP status | 5 checksum
             6 storage/I-O | 7 server/agent error | 130 interrupted
 """
 
-COMMANDS = ("serve", "agent", "send", "jobs")
+COMMANDS = ("serve", "agent", "send", "jobs", "token", "config")
 
 
 # ----------------------------------------------------------------- shared bits
@@ -161,6 +171,125 @@ def _client(args: argparse.Namespace) -> ServerClient:
     )
 
 
+# ------------------------------------------------------------------ config file
+
+
+def _bool(value: str) -> bool:
+    low = value.strip().lower()
+    if low in ("1", "true", "yes", "on"):
+        return True
+    if low in ("0", "false", "no", "off"):
+        return False
+    raise ValueError(f"not a boolean: {value!r}")
+
+
+def _list(value: str) -> List[str]:
+    return [item.strip() for item in value.split(",") if item.strip()]
+
+
+Spec = Dict[str, Tuple[str, Callable[[str], Any]]]
+
+DOWNLOAD_SPEC: Spec = {
+    "path": ("path", str), "connections": ("connections", int), "retries": ("retries", int),
+    "timeout": ("timeout", float), "limit_rate": ("limit_rate", str), "user_agent": ("user_agent", str),
+    "proxy": ("proxy", str), "ca_cert": ("ca_cert", str),
+}
+CLIENT_SPEC: Spec = {
+    "server": ("server", str), "verify": ("server_no_check_certificate", lambda v: not _bool(v)),
+    "ca_cert": ("server_ca_cert", str),
+}
+_COMMON_SERVER: Spec = {
+    "host": ("host", str), "port": ("port", int), "state_dir": ("state_dir", str),
+    "max_jobs": ("max_jobs", int), "allowed_host": ("allowed_host", _list),
+    "tls_cert": ("tls_cert", str), "tls_key": ("tls_key", str), "ui": ("no_ui", lambda v: not _bool(v)),
+}
+SERVE_SPEC: Spec = dict(_COMMON_SERVER, **{
+    "root": ("root", str), "connections": ("connections", int), "retries": ("retries", int),
+    "limit_rate": ("limit_rate", str), "deny_private_networks": ("deny_private_networks", _bool),
+})
+AGENT_SPEC: Spec = dict(_COMMON_SERVER, **{
+    "upstream": ("upstream", str), "upstream_token": ("upstream_token", str),
+    "upstream_verify": ("upstream_no_check_certificate", lambda v: not _bool(v)),
+    "upstream_ca_cert": ("upstream_ca", str),
+})
+
+
+def _apply_config(parser: argparse.ArgumentParser, section: str, spec: Spec) -> None:
+    """Make config-file values the parser defaults (command line still wins)."""
+    values = load_config().get(section, {})
+    unknown = sorted(set(values) - set(spec))
+    if unknown:
+        raise ConfigError(
+            f"unknown option(s) in [{section}] of {config_file_path()}: {', '.join(unknown)}"
+        )
+    defaults: Dict[str, Any] = {}
+    for key, raw in values.items():
+        dest, convert = spec[key]
+        if raw == "":
+            continue
+        try:
+            defaults[dest] = convert(raw)
+        except ValueError:
+            raise ConfigError(f"invalid value for [{section}] {key} in {config_file_path()}: {raw!r}") from None
+    parser.set_defaults(**defaults)
+
+
+def _config_main(argv: List[str]) -> int:
+    p = argparse.ArgumentParser(prog="pywget config", description="Manage the optional config file.")
+    p.add_argument("action", choices=["path", "init", "show"])
+    p.add_argument("--force", action="store_true", help="init: overwrite an existing file")
+    args = p.parse_args(argv)
+    path = config_file_path()
+    if args.action == "path":
+        print(path)
+    elif args.action == "init":
+        print(write_default_config(args.force))
+    else:
+        if not path.exists():
+            print(f"(no config file; create one with: pywget config init)  [{path}]", file=sys.stderr)
+            return 0
+        shown = []
+        for line in path.read_text(encoding="utf-8").splitlines():
+            key = line.split("=", 1)[0].strip().lower()
+            shown.append(f"{key} = ********" if key == "upstream_token" and "=" in line else line)
+        print("\n".join(shown))
+    return 0
+
+
+def _token_main(argv: List[str]) -> int:
+    p = argparse.ArgumentParser(
+        prog="pywget token",
+        description="Manage the API token (stored in the token file, mode 0600).",
+    )
+    p.add_argument("action", choices=["show", "new", "set", "path"])
+    p.add_argument("value", nargs="?", help="set: the token to use ('-' = read from stdin)")
+    p.add_argument("--length", type=int, default=32, metavar="BYTES", help="new: random bytes (default 32)")
+    args = p.parse_args(argv)
+    if args.action == "path":
+        print(token_path())
+        return 0
+    if args.action == "show":
+        token, source = resolve_token()
+        if not token:
+            print("no token yet: it is generated the first time you run 'pywget serve' (or run 'pywget token new')",
+                  file=sys.stderr)
+            return 1
+        print(f"token source: {source}", file=sys.stderr)
+        print(token)
+        return 0
+    if args.action == "new":
+        token = generate_token(args.length)
+    else:
+        if not args.value:
+            p.error("'set' needs a value (or '-' to read stdin)")
+        token = (sys.stdin.readline() if args.value == "-" else args.value).strip()
+        validate_token(token)
+    path = save_token(token)
+    print(token)
+    print(f"saved to {path}; restart 'pywget serve' / 'pywget agent' to apply it", file=sys.stderr)
+    return 0
+
+
 # ------------------------------------------------------------- direct download
 
 
@@ -223,6 +352,7 @@ def _build_options(args: argparse.Namespace, progress: Optional[ProgressBar]) ->
 
 def _download_main(argv: List[str]) -> int:
     parser = build_parser()
+    _apply_config(parser, "download", DOWNLOAD_SPEC)
     args = parser.parse_args(argv)
     try:
         urls = _expand_inputs(args.urls, args.input_file)
@@ -300,12 +430,19 @@ def _job_specs(args: argparse.Namespace, urls: List[str]) -> List[Dict[str, Any]
     return specs
 
 
+def _clip(text: str, width: int) -> str:
+    return text if len(text) <= width else text[: max(width - 2, 1)] + ".."
+
+
 def _fmt_job(job: Dict[str, Any]) -> str:
+    cols = shutil.get_terminal_size((100, 24)).columns
     pct = f"{job['percent']:5.1f}%" if job.get("percent") is not None else "   --%"
     speed = f"{format_size(job['speed'])}/s" if job.get("status") == "running" else ""
-    label = job.get("result_path") or job.get("filename") or job["url"]
-    err = f"  ! {job['error']}" if job.get("error") else ""
-    return f"{job['id']}  {job['status']:<9} {pct} {speed:>11}  {label}{err}"
+    prefix = f"{job['id']}  {job['status']:<9} {pct} {speed:>11}  "
+    line = prefix + _clip(job.get("name") or job["url"], max(20, cols - len(prefix) - 1))
+    if job.get("error"):
+        line += "\n    ! " + _clip(job["error"], max(20, cols - 8))
+    return line
 
 
 def _forward(args: argparse.Namespace, urls: List[str], parser: argparse.ArgumentParser) -> int:
@@ -329,7 +466,7 @@ def _forward(args: argparse.Namespace, urls: List[str], parser: argparse.Argumen
                 last = [""]
 
                 def show(j: Dict[str, Any], last: List[str] = last) -> None:
-                    line = _fmt_job(j)
+                    line = _fmt_job(j).split("\n")[0]
                     if not quiet and line != last[0] and sys.stderr.isatty():
                         sys.stderr.write("\r" + line + " " * max(0, len(last[0]) - len(line)))
                         sys.stderr.flush()
@@ -368,6 +505,7 @@ def _send_main(argv: List[str]) -> int:
     p.add_argument("-q", "--quiet", action="store_true")
     p.add_argument("-v", "--verbose", action="store_true")
     p.add_argument("--json", action="store_true", help="print jobs as JSON on stdout")
+    _apply_config(p, "client", CLIENT_SPEC)
     args = p.parse_args(argv)
     try:
         urls = _expand_inputs(args.urls, args.input_file)
@@ -392,6 +530,7 @@ def _jobs_main(argv: List[str]) -> int:
     p.add_argument("--status", help="filter for 'list' (queued, running, paused, completed, failed, cancelled)")
     p.add_argument("--json", action="store_true")
     _add_server_args(p)
+    _apply_config(p, "client", CLIENT_SPEC)
     args = p.parse_args(argv)
     if args.action not in ("list", "stats") and not args.job_id:
         p.error(f"'{args.action}' needs a job id")
@@ -436,7 +575,7 @@ def _serve_main(argv: List[str], agent: bool) -> int:
     p.add_argument("--port", type=int, default=DEFAULT_AGENT_PORT if agent else DEFAULT_PORT, help="TCP port")
     p.add_argument("--token", help="API token (default: $PYWGET_TOKEN, token file, or generated)")
     if agent:
-        p.add_argument("--upstream", required=True, metavar="URL", help="server to forward links to")
+        p.add_argument("--upstream", metavar="URL", help="server to forward links to (or 'upstream' in [agent])")
         p.add_argument("--upstream-token", metavar="TOKEN", help="token of the upstream server")
         p.add_argument("--upstream-no-check-certificate", action="store_true")
         p.add_argument("--upstream-ca-cert", metavar="FILE")
@@ -456,13 +595,16 @@ def _serve_main(argv: List[str], agent: bool) -> int:
     p.add_argument("--tls-key", metavar="FILE")
     p.add_argument("--no-ui", action="store_true", help="disable the web UI")
     p.add_argument("-v", "--verbose", action="store_true")
+    _apply_config(p, "agent" if agent else "serve", AGENT_SPEC if agent else SERVE_SPEC)
     args = p.parse_args(argv)
+    if agent and not args.upstream:
+        p.error("--upstream is required (or set 'upstream' in the [agent] section of the config file)")
     _setup_logging(args.verbose)
     _install_signals()
     try:
-        token = load_token(args.token, create=True)
+        token, token_source = resolve_token(args.token, create=True)
         kwargs: Dict[str, Any] = dict(
-            host=args.host, port=args.port, token=token or "", max_jobs=args.max_jobs,
+            host=args.host, port=args.port, token=token or "", token_source=token_source, max_jobs=args.max_jobs,
             allowed_hosts=args.allowed_host, tls_cert=args.tls_cert, tls_key=args.tls_key, ui=not args.no_ui,
         )
         if args.state_dir:
@@ -500,14 +642,26 @@ def main(argv: Optional[List[str]] = None) -> int:
     args = list(sys.argv[1:] if argv is None else argv)
     if args and args[0] in COMMANDS:
         command, rest = args[0], args[1:]
-        if command == "serve":
-            return _serve_main(rest, agent=False)
-        if command == "agent":
-            return _serve_main(rest, agent=True)
-        if command == "send":
-            return _send_main(rest)
-        return _jobs_main(rest)
-    return _download_main(args)
+        try:
+            if command == "serve":
+                return _serve_main(rest, agent=False)
+            if command == "agent":
+                return _serve_main(rest, agent=True)
+            if command == "send":
+                return _send_main(rest)
+            if command == "token":
+                return _token_main(rest)
+            if command == "config":
+                return _config_main(rest)
+            return _jobs_main(rest)
+        except ConfigError as exc:
+            print(f"pywget: error: {exc}", file=sys.stderr)
+            return exc.exit_code
+    try:
+        return _download_main(args)
+    except ConfigError as exc:
+        print(f"pywget: error: {exc}", file=sys.stderr)
+        return exc.exit_code
 
 
 if __name__ == "__main__":  # pragma: no cover

pywget-2.0.6/pywget/config.py

@@ -0,0 +1,220 @@
+"""Configuration helpers: config dir, API token, server configuration."""
+
+from __future__ import annotations
+
+import configparser
+import os
+import re
+import secrets
+import sys
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Any, Dict, List, Optional, Tuple
+
+from .errors import ConfigError
+
+DEFAULT_HOST = "127.0.0.1"
+DEFAULT_PORT = 8765
+DEFAULT_AGENT_PORT = 8766
+LOOPBACK_HOSTS = ("127.0.0.1", "localhost", "::1")
+
+
+def config_dir() -> Path:
+    override = os.environ.get("PYWGET_CONFIG_DIR")
+    if override:
+        return Path(override).expanduser()
+    if sys.platform == "win32":
+        base = os.environ.get("APPDATA") or str(Path.home() / "AppData" / "Roaming")
+        return Path(base) / "pywget"
+    base = os.environ.get("XDG_CONFIG_HOME") or str(Path.home() / ".config")
+    return Path(base) / "pywget"
+
+
+def token_path() -> Path:
+    return config_dir() / "token"
+
+
+def config_file_path() -> Path:
+    override = os.environ.get("PYWGET_CONFIG")
+    return Path(override).expanduser() if override else config_dir() / "config.ini"
+
+
+_TOKEN_RE = re.compile(r"[A-Za-z0-9._~+/=-]{16,256}")
+
+
+def validate_token(token: str) -> str:
+    """A token is 16-256 characters of A-Z a-z 0-9 . _ ~ + / = - (safe in HTTP headers)."""
+    if not _TOKEN_RE.fullmatch(token or ""):
+        raise ConfigError(
+            "invalid token: use 16-256 characters from A-Z a-z 0-9 . _ ~ + / = - "
+            "(generate one with: pywget token new)"
+        )
+    return token
+
+
+def generate_token(nbytes: int = 32) -> str:
+    return secrets.token_urlsafe(max(16, nbytes))
+
+
+def _write_private(path: Path, text: str) -> None:
+    path.parent.mkdir(parents=True, exist_ok=True)
+    tmp = path.with_name(path.name + ".tmp")
+    fd = os.open(str(tmp), os.O_CREAT | os.O_TRUNC | os.O_WRONLY, 0o600)
+    with os.fdopen(fd, "w", encoding="utf-8") as handle:
+        handle.write(text)
+    os.replace(tmp, path)
+
+
+def save_token(token: str) -> Path:
+    validate_token(token)
+    path = token_path()
+    _write_private(path, token + "\n")
+    return path
+
+
+def resolve_token(explicit: Optional[str] = None, create: bool = False) -> Tuple[Optional[str], str]:
+    """Return (token, source). Order: explicit > $PYWGET_TOKEN > token file > generated."""
+    if explicit:
+        return explicit.strip(), "command line"
+    env = os.environ.get("PYWGET_TOKEN")
+    if env:
+        return env.strip(), "$PYWGET_TOKEN"
+    path = token_path()
+    try:
+        text = path.read_text(encoding="utf-8").strip()
+        if text:
+            return text, str(path)
+    except OSError:
+        pass
+    if not create:
+        return None, ""
+    token = generate_token()
+    path.parent.mkdir(parents=True, exist_ok=True)
+    try:
+        fd = os.open(str(path), os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
+    except FileExistsError:  # lost a race with another process
+        return path.read_text(encoding="utf-8").strip(), str(path)
+    with os.fdopen(fd, "w", encoding="utf-8") as handle:
+        handle.write(token + "\n")
+    return token, f"{path} (generated)"
+
+
+def load_token(explicit: Optional[str] = None, create: bool = False) -> Optional[str]:
+    return resolve_token(explicit, create)[0]
+
+
+DEFAULT_CONFIG = """\
+# pywget configuration file (INI).
+# Precedence: command line  >  environment  >  this file  >  built-in defaults.
+# Everything below is commented out: uncomment and edit what you need.
+# Booleans: true / false.  Lists: comma separated.  The API token is NOT stored here
+# (see: pywget token show | new | set).
+
+[download]
+# options for direct downloads:  pywget URL
+;path = ~/Downloads
+;connections = 4
+;retries = 5
+;timeout = 30
+;limit_rate = 2M
+;user_agent =
+;proxy =
+;ca_cert =
+
+[client]
+# options for: pywget send / jobs   (the server to talk to)
+;server = http://127.0.0.1:8765
+;verify = true
+;ca_cert =
+
+[serve]
+# options for: pywget serve
+;host = 127.0.0.1
+;port = 8765
+;root = ~/Downloads/pywget
+;state_dir =
+;max_jobs = 3
+;connections = 4
+;retries = 5
+;limit_rate =
+;deny_private_networks = false
+;allowed_host =
+;tls_cert =
+;tls_key =
+;ui = true
+
+[agent]
+# options for: pywget agent
+;host = 127.0.0.1
+;port = 8766
+;upstream = https://nas:8765
+;upstream_token =
+;upstream_verify = true
+;upstream_ca_cert =
+;max_jobs = 32
+;state_dir =
+;allowed_host =
+;tls_cert =
+;tls_key =
+;ui = true
+"""
+
+
+def load_config() -> Dict[str, Dict[str, str]]:
+    path = config_file_path()
+    if not path.exists():
+        return {}
+    parser = configparser.ConfigParser(interpolation=None)
+    try:
+        with open(path, encoding="utf-8") as handle:
+            parser.read_file(handle)
+    except (configparser.Error, OSError, UnicodeDecodeError) as exc:
+        raise ConfigError(f"cannot read config file {path}: {exc}") from None
+    return {name: dict(parser.items(name)) for name in parser.sections()}
+
+
+def write_default_config(force: bool = False) -> Path:
+    path = config_file_path()
+    if path.exists() and not force:
+        raise ConfigError(f"{path} already exists (use --force to overwrite)")
+    _write_private(path, DEFAULT_CONFIG)
+    return path
+
+
+@dataclass
+class ServerConfig:
+    host: str = DEFAULT_HOST
+    port: int = DEFAULT_PORT
+    token: str = ""
+    root: Path = field(default_factory=lambda: Path.home() / "Downloads" / "pywget")
+    state_dir: Path = field(default_factory=lambda: config_dir() / "server")
+    max_jobs: int = 3
+    connections: int = 4
+    retries: int = 5
+    limit_rate: Any = 0
+    deny_private: bool = False
+    allowed_hosts: List[str] = field(default_factory=list)
+    tls_cert: Optional[str] = None
+    tls_key: Optional[str] = None
+    ui: bool = True
+    upstream: Optional[str] = None
+    upstream_token: Optional[str] = None
+    upstream_verify: bool = True
+    upstream_ca: Optional[str] = None
+    token_source: str = ""
+
+    def __post_init__(self) -> None:
+        self.root = Path(self.root).expanduser()
+        self.state_dir = Path(self.state_dir).expanduser()
+        try:
+            validate_token(self.token)
+        except ConfigError as exc:
+            raise ValueError(str(exc)) from None
+        if not 1 <= self.max_jobs <= 256:
+            raise ValueError("max_jobs must be between 1 and 256")
+        if bool(self.tls_cert) != bool(self.tls_key):
+            raise ValueError("--tls-cert and --tls-key must be given together")
+
+    @property
+    def is_agent(self) -> bool:
+        return bool(self.upstream)

{pywget-2.0.4 → pywget-2.0.6}/pywget/core.py

@@ -54,6 +54,7 @@ from .utils import (
     parse_checksum,
     parse_size,
     sanitize_filename,
+    short_url,
     unique_path,
 )
 
@@ -191,6 +192,36 @@ class _Worker(threading.Thread):
             self.error = exc
 
 
+class _ConcurrencyGate:
+    """Limits simultaneous segment requests; shrinks when the server answers 429."""
+
+    def __init__(self, limit: int) -> None:
+        self.limit = max(1, limit)
+        self._active = 0
+        self._cond = threading.Condition()
+
+    def acquire(self, cancel: threading.Event) -> bool:
+        with self._cond:
+            while self._active >= self.limit:
+                if cancel.is_set():
+                    return False
+                self._cond.wait(0.2)
+            self._active += 1
+            return True
+
+    def release(self) -> None:
+        with self._cond:
+            self._active -= 1
+            self._cond.notify_all()
+
+    def shrink(self) -> bool:
+        with self._cond:
+            if self.limit > 1:
+                self.limit -= 1
+                return True
+            return False
+
+
 def _is_retryable(exc: BaseException) -> bool:
     if isinstance(exc, NetworkError):
         return exc.retryable
@@ -252,6 +283,7 @@ class Downloader:
         self._last_report = 0.0
         self._tracker: Optional[Tracker] = None
         self.partial: Optional[Tuple[Path, Path]] = None  # (.part, .pywget) of the current download
+        self._gate = _ConcurrencyGate(self.options.connections)
         self._opener = self._build_opener()
 
     # ------------------------------------------------------------------ public
@@ -383,11 +415,11 @@ class Downloader:
         except urllib.error.URLError as exc:
             if isinstance(exc.reason, ssl.SSLCertVerificationError):
                 raise NetworkError(
-                    f"TLS certificate verification failed for {url}: {exc.reason} "
+                    f"TLS certificate verification failed for {short_url(url)}: {exc.reason} "
                     "(use --no-check-certificate or --ca-cert)",
                     retryable=False,
                 ) from None
-            raise NetworkError(f"{url}: {exc.reason}") from None
+            raise NetworkError(f"{short_url(url)}: {exc.reason}") from None
 
     def _probe(self, url: str) -> RemoteInfo:
         try:
@@ -481,6 +513,7 @@ class Downloader:
         else:
             connections = len(segments)
             log.info("resuming %s (%s already on disk)", part.name, format_size(sum(s[2] for s in segments)))
+        self._gate = _ConcurrencyGate(connections)
         tracker.set(sum(s[2] for s in segments))
         todo = [s for s in segments if s[2] < s[1] - s[0] + 1]
         workers = [
@@ -514,10 +547,14 @@ class Downloader:
     ) -> None:
         attempt = 0
         read_size = self._read_size()
+        gate = self._gate
         while not self._cancel.is_set():
             start = seg[0] + seg[2]
             if start > seg[1]:
                 return
+            if not gate.acquire(self._cancel):
+                return
+            held = True
             before = seg[2]
             try:
                 with closing(self._open(url, {"Range": f"bytes={start}-{seg[1]}"})) as resp:
@@ -539,6 +576,8 @@ class Downloader:
                             seg[2] += len(chunk)
                             tracker.add(len(chunk))
             except Exception as exc:  # noqa: BLE001
+                gate.release()
+                held = False
                 if self._cancel.is_set():
                     return
                 if not _is_retryable(exc):
@@ -546,9 +585,15 @@ class Downloader:
                 if seg[2] > before:
                     attempt = 0
                 attempt += 1
+                if isinstance(exc, HTTPStatusError) and exc.status == 429 and gate.shrink():
+                    log.warning("server limits parallel connections: reducing to %d", gate.limit)
+                    attempt -= 1  # a concurrency 429 is not a failed attempt
                 if attempt > self.options.retries:
                     raise self._final_error(exc, "segment") from (None if isinstance(exc, PywgetError) else exc)
-                self._sleep_backoff(attempt, exc, f"segment {seg[0]}-{seg[1]}")
+                self._sleep_backoff(max(attempt, 1), exc, f"segment {seg[0]}-{seg[1]}")
+            finally:
+                if held:
+                    gate.release()
 
     def _run_stream(
         self, url: str, info: RemoteInfo, part: Path, tracker: Tracker, limiter: RateLimiter

{pywget-2.0.4 → pywget-2.0.6}/pywget/errors.py

@@ -46,8 +46,10 @@ class HTTPStatusError(NetworkError):
         reason: str = "",
         retry_after: Optional[float] = None,
     ) -> None:
+        from .utils import short_url  # lazy: utils imports this module
+
         text = f"HTTP {status} {reason}".strip()
-        super().__init__(f"{text} for {url}", retryable=status in RETRY_STATUS)
+        super().__init__(f"{text} for {short_url(url)}", retryable=status in RETRY_STATUS)
         self.status = status
         self.url = url
         self.retry_after = retry_after
@@ -74,3 +76,9 @@ class ServerError(PywgetError):
         super().__init__(message)
         self.status = status
         self.retryable = retryable
+
+
+class ConfigError(PywgetError):
+    """Invalid configuration file or token."""
+
+    exit_code = 2

{pywget-2.0.4 → pywget-2.0.6}/pywget/jobs.py

@@ -18,7 +18,14 @@ from .config import ServerConfig
 from .core import Downloader, Options
 from .errors import DownloadCancelled, InvalidURLError, PywgetError, ServerError
 from .progress import ProgressInfo
-from .utils import RateLimiter, build_headers, normalize_url, parse_checksum, parse_size
+from .utils import (
+    RateLimiter,
+    build_headers,
+    filename_from_url,
+    normalize_url,
+    parse_checksum,
+    parse_size,
+)
 
 log = logging.getLogger("pywget.jobs")
 
@@ -82,6 +89,8 @@ class Job:
         data = self.to_dict()
         for name in _SECRET_FIELDS:
             data.pop(name, None)
+        shown = self.result_path.replace("\\", "/").rsplit("/", 1)[-1] if self.result_path else None
+        data["name"] = self.filename or shown or filename_from_url(self.url) or self.url
         data["has_auth"] = bool(self.username)
         data["has_cookie"] = bool(self.cookie)
         data["header_names"] = sorted(self.headers)

{pywget-2.0.4 → pywget-2.0.6}/pywget/server.py

@@ -307,6 +307,8 @@ def run_server(cfg: ServerConfig) -> int:
     import sys
 
     print(f"pywget {kind} listening on {handle.url}", file=sys.stderr)
+    if cfg.token_source:
+        print(f"api token: {cfg.token_source}  (show it with: pywget token show)", file=sys.stderr)
     if cfg.is_agent:
         print(f"forwarding links to {cfg.upstream}", file=sys.stderr)
     else:

{pywget-2.0.4 → pywget-2.0.6}/pywget/utils.py

@@ -197,3 +197,14 @@ def build_headers(
         if not name or any(c in name + value for c in "\r\n\x00") or ":" in name:
             raise ValueError(f"invalid header: {name!r}")
     return merged
+
+
+def short_url(url: str, limit: int = 70) -> str:
+    """``host/../file.zip`` - no query string (it often holds signed tokens), bounded length."""
+    parts = urlsplit(url)
+    name = unquote(parts.path.rsplit("/", 1)[-1]) or "/"
+    host = parts.hostname or ""
+    room = max(limit - len(host) - 5, 12)
+    if len(name) > room:
+        name = name[: room - 12] + ".." + name[-10:]
+    return f"{host}/../{name}" if parts.path.count("/") > 1 else f"{host}/{name}".rstrip("/")

{pywget-2.0.4 → pywget-2.0.6}/pywget/webui.py

@@ -35,7 +35,7 @@ function cell(tr,text,cls){const td=document.createElement('td');td.textContent=
 async function refresh(){try{const[s,j]=await Promise.all([api('GET','/api/v1/stats'),api('GET','/api/v1/jobs')]);
 $('mode').textContent=s.mode;$('stats').textContent=fmt(s.speed)+'/s';$('msg').textContent='';const body=$('rows');body.replaceChildren();
 for(const x of j.jobs.slice().reverse()){const tr=document.createElement('tr');
-const name=x.result_path?x.result_path.split(/[\\\\/]/).pop():(x.filename||x.url);cell(tr,name,'n').title=x.url;
+const name=x.name||x.url;cell(tr,name,'n').title=x.url;
 cell(tr,x.status+(x.error?' - '+x.error:''));const td=document.createElement('td');const pr=document.createElement('progress');pr.max=100;pr.value=x.percent||0;td.appendChild(pr);td.append(' '+(x.percent!=null?x.percent+'%':''));tr.appendChild(td);
 cell(tr,x.status==='running'?fmt(x.speed)+'/s':'-','mut');const act=document.createElement('td');
 const btn=(label,fn)=>{const b=document.createElement('button');b.textContent=label;b.onclick=async()=>{try{await fn();refresh()}catch(e){$('msg').textContent=e.message}};act.appendChild(b)};

{pywget-2.0.4 → pywget-2.0.6/pywget.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pywget
-Version: 2.0.4
+Version: 2.0.6
 Summary: Robust, resumable, multi-connection wget alternative - CLI and Python library, zero dependencies
 Author-email: Hadi Cahyadi <cumulus13@gmail.com>
 Maintainer-email: cumulus13 <cumulus13@gmail.com>
@@ -231,6 +231,58 @@ Errors derive from `pywget.PywgetError` (`InvalidURLError`, `NetworkError`,
 `HTTPStatusError`, `ChecksumError`, `StorageError`, `DownloadCancelled`).
 A `Downloader` runs one download at a time - use one instance per thread.
 
+## Configuration and API token
+
+### Config file (optional)
+
+There is no config file unless you create one; every option also has a command-line flag.
+
+```bash
+pywget config path     # where it lives
+pywget config init     # write a fully commented template there (mode 0600)
+pywget config show
+```
+
+| OS | Default location |
+|----|------------------|
+| Windows | `%APPDATA%\pywget\config.ini` |
+| Linux / macOS | `~/.config/pywget/config.ini` (`$XDG_CONFIG_HOME` respected) |
+
+Override the directory with `PYWGET_CONFIG_DIR` or the file with `PYWGET_CONFIG`.
+Precedence: **command line > environment > config file > built-in defaults**.
+Sections: `[download]`, `[client]`, `[serve]`, `[agent]` (the generated template lists every key).
+Unknown keys or bad values are reported with the file path and exit code 2.
+
+```ini
+[serve]
+root = C:\DOWNLOADS
+max_jobs = 3
+connections = 2
+
+[client]
+server = http://127.0.0.1:8765
+```
+
+Job queue state lives in `<config dir>/server/jobs.json` (`agent/` for an agent); change it with `state_dir`.
+
+### API token
+
+The server and agent **always** require a token.
+
+- **Default:** on the first `pywget serve` / `pywget agent` a random 256-bit token is generated and saved to `<config dir>/token` (mode 0600). It is reused on every start, and local `send` / `jobs` commands read it automatically.
+- **Show / create / set:**
+
+```bash
+pywget token show            # print the current token and where it comes from
+pywget token new             # generate and save a new one (restart the server afterwards)
+pywget token set "my-own-secret-123"   # or: echo -n ... | pywget token set -
+pywget token path
+```
+
+- **Override order:** `--token` > `$PYWGET_TOKEN` > token file.
+- **Format:** not arbitrary text - 16 to 256 characters from `A-Z a-z 0-9 . _ ~ + / = -` (safe inside an HTTP header). Weak or malformed tokens are rejected at startup.
+- **Remote clients:** copy the token to the other machine (`pywget token set -` there, or `export PYWGET_TOKEN=...`).
+
 ## Server mode and agent
 
 Run pywget as a long-lived download manager and push links to it from anywhere - the
@@ -573,7 +625,7 @@ classDiagram
 | `pywget.server` | HTTP API, SSE, web UI hosting (`ServerHandle`, `run_server`) |
 | `pywget.jobs` | `Job`, `JobManager`, persistence, `LocalExecutor` / `RelayExecutor` |
 | `pywget.client` | `ServerClient` (used by `send`, `jobs` and the agent) |
-| `pywget.config` | Token file, config dir, `ServerConfig` |
+| `pywget.config` | Config file, token file and validation, `ServerConfig` |
 | `pywget.clipboard` | `c` input: clipboard and URL extraction |
 | `pywget.webui` | Embedded single-page web UI |
 | `pywget.wget` | Backward-compatible `wget.download()` |
@@ -582,6 +634,8 @@ classDiagram
 
 - **Probing:** one `Range: bytes=0-0` request yields size, range support, ETag, `Last-Modified` and `Content-Disposition`.
 - **Segments:** a file is split into `min(connections, size / 2 MiB)` parts; small files use one connection.
+- **Rate-limited servers:** on HTTP 429 the number of simultaneous segment requests is reduced automatically (down to 1) instead of failing the job; failed jobs keep partial data, so `pywget jobs retry ID` resumes.
+- **Output hygiene:** messages and job lists show `host/../file` only - query strings (often signed tokens) are never printed.
 - **Existing file:** same size means skipped; different size is saved as `name.1` (wget style) unless `--overwrite`.
 - **Partial files:** stored as `name.part` + `name.pywget`; removed on success. Changed remote file (size/ETag/mtime) restarts cleanly.
 - **Redirects:** only http/https; `Authorization`/`Cookie` are dropped when the origin changes.

{pywget-2.0.4 → pywget-2.0.6}/pywget.egg-info/SOURCES.txt

@@ -23,6 +23,7 @@ pywget.egg-info/dependency_links.txt
 pywget.egg-info/entry_points.txt
 pywget.egg-info/requires.txt
 pywget.egg-info/top_level.txt
+tests/test_config.py
 tests/test_download.py
 tests/test_server.py
 tests/test_utils.py

pywget-2.0.6/tests/test_config.py

@@ -0,0 +1,138 @@
+import pytest
+from conftest import DATA
+
+from pywget import Downloader, Options
+from pywget.cli import _fmt_job, build_parser, main
+from pywget.config import ServerConfig, load_config, resolve_token, token_path, validate_token
+from pywget.errors import ConfigError, HTTPStatusError
+from pywget.utils import short_url
+
+
+@pytest.fixture
+def cfgdir(tmp_path, monkeypatch):
+    monkeypatch.setenv("PYWGET_CONFIG_DIR", str(tmp_path / "cfg"))
+    monkeypatch.delenv("PYWGET_TOKEN", raising=False)
+    monkeypatch.delenv("PYWGET_CONFIG", raising=False)
+    monkeypatch.delenv("PYWGET_SERVER", raising=False)
+    return tmp_path / "cfg"
+
+
+def test_token_auto_generated_once_and_reused(cfgdir):
+    assert resolve_token() == (None, "")
+    token, source = resolve_token(create=True)
+    assert len(token) >= 40 and "generated" in source
+    assert token_path().read_text().strip() == token
+    assert resolve_token(create=True)[0] == token  # reused, not regenerated
+
+
+def test_token_precedence(cfgdir, monkeypatch):
+    file_token, _ = resolve_token(create=True)
+    monkeypatch.setenv("PYWGET_TOKEN", "e" * 20)
+    assert resolve_token()[0] == "e" * 20
+    assert resolve_token("x" * 20)[0] == "x" * 20
+    assert file_token not in ("e" * 20, "x" * 20)
+
+
+@pytest.mark.parametrize("bad", ["short", "has space " * 3, "ünicode-" * 4, "a" * 300, ""])
+def test_token_validation_rejects(bad):
+    with pytest.raises(ConfigError):
+        validate_token(bad)
+    with pytest.raises(ValueError):
+        ServerConfig(token=bad)
+
+
+def test_token_accepts_urlsafe():
+    assert validate_token("Abc-123_xyz.~+/=0123456789")
+
+
+def test_token_cli(cfgdir, capsys):
+    assert main(["token", "show"]) == 1
+    assert main(["token", "new"]) == 0
+    new = capsys.readouterr().out.strip().splitlines()[0]
+    assert main(["token", "show"]) == 0
+    assert capsys.readouterr().out.strip() == new
+    assert main(["token", "set", "tooshort"]) == 2
+    assert main(["token", "set", "my-own-secret-token-123"]) == 0
+    assert token_path().read_text().strip() == "my-own-secret-token-123"
+    assert main(["token", "path"]) == 0
+
+
+def test_config_init_show_and_defaults(cfgdir, capsys):
+    assert main(["config", "init"]) == 0
+    assert (cfgdir / "config.ini").exists()
+    assert main(["config", "init"]) == 2  # exists
+    assert main(["config", "init", "--force"]) == 0
+    assert all(not values for values in load_config().values())  # template is fully commented out
+    path = cfgdir / "config.ini"
+    path.write_text("[download]\nconnections = 2\nretries = 9\nlimit_rate = 1M\n[client]\nserver = http://nas:1\n"
+                    "verify = false\n")
+    from pywget.cli import CLIENT_SPEC, DOWNLOAD_SPEC, _apply_config
+
+    parser = build_parser()
+    _apply_config(parser, "download", DOWNLOAD_SPEC)
+    args = parser.parse_args(["http://x/y"])
+    assert (args.connections, args.retries, args.limit_rate) == (2, 9, "1M")
+    assert parser.parse_args(["http://x/y", "-n", "7"]).connections == 7  # command line wins
+    _apply_config(parser, "client", CLIENT_SPEC)
+    args = parser.parse_args(["http://x/y"])
+    assert args.server == "http://nas:1" and args.server_no_check_certificate is True
+    assert main(["config", "show"]) == 0
+
+
+def test_config_errors(cfgdir, capsys):
+    cfgdir.mkdir(parents=True)
+    (cfgdir / "config.ini").write_text("[download]\nbogus = 1\n")
+    assert main(["http://x/y", "-q"]) == 2
+    assert "bogus" in capsys.readouterr().err
+    (cfgdir / "config.ini").write_text("[download]\nconnections = many\n")
+    assert main(["http://x/y", "-q"]) == 2
+    (cfgdir / "config.ini").write_text("not an ini file")
+    assert main(["http://x/y", "-q"]) == 2
+
+
+def test_config_serve_section_and_agent_upstream(cfgdir, capsys):
+    cfgdir.mkdir(parents=True)
+    (cfgdir / "config.ini").write_text("[serve]\nmax_jobs = 5\nui = false\nallowed_host = a, b\n")
+    import argparse
+
+    from pywget.cli import SERVE_SPEC, _apply_config
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--max-jobs", dest="max_jobs", type=int, default=3)
+    parser.add_argument("--no-ui", dest="no_ui", action="store_true")
+    parser.add_argument("--allowed-host", dest="allowed_host", action="append", default=[])
+    _apply_config(parser, "serve", {k: v for k, v in SERVE_SPEC.items() if k in ("max_jobs", "ui", "allowed_host")})
+    args = parser.parse_args([])
+    assert (args.max_jobs, args.no_ui, args.allowed_host) == (5, True, ["a", "b"])
+
+
+def test_agent_requires_upstream(cfgdir):
+    with pytest.raises(SystemExit) as err:
+        main(["agent"])
+    assert err.value.code == 2
+
+
+def test_short_url_hides_query_and_truncates():
+    url = "https://nw28.seedr.cc/get_zip/123/" + "Very%20Long%20Name-" * 8 + ".zip?st=SECRET&e=1"
+    short = short_url(url)
+    assert "SECRET" not in short and len(short) <= 75 and short.endswith(".zip")
+    assert short_url("http://h/file.bin") == "h/file.bin"
+
+
+def test_http_error_message_has_no_query(server, tmp_path):
+    with pytest.raises(HTTPStatusError) as err:
+        Downloader(Options(output_dir=str(tmp_path), retries=0)).fetch(server + "/nope.zip?st=SECRET")
+    assert "SECRET" not in str(err.value) and "nope.zip" in str(err.value)
+
+
+def test_fmt_job_is_one_line_plus_error(monkeypatch):
+    monkeypatch.setattr("shutil.get_terminal_size", lambda fallback=(100, 24): __import__("os").terminal_size((100, 24)))
+    job = {"id": "abc", "status": "failed", "percent": 24.3, "speed": 0, "url": "http://h/" + "x" * 500,
+           "name": "n" * 500, "error": "boom " * 100}
+    lines = _fmt_job(job).split("\n")
+    assert len(lines) == 2 and all(len(line) <= 100 for line in lines)
+
+
+def test_adaptive_connections_on_429(server, tmp_path):
+    res = Downloader(Options(output_dir=str(tmp_path), connections=3, backoff=0.01, max_backoff=0.05,
+                             retries=2)).fetch(server + "/limited")
+    assert res.path.read_bytes() == DATA

{pywget-2.0.4 → pywget-2.0.6}/tests/test_server.py

@@ -228,8 +228,12 @@ def test_agent_waits_for_offline_upstream(tmp_path, server):
     with ServerHandle(agent_cfg) as agent:
         ac = client_for(agent, "a" * 40)
         job = ac.submit({"url": server + "/small.txt"})[0]
-        time.sleep(1.5)
+        # Windows needs ~2s per refused connection attempt, so poll instead of a fixed sleep
+        deadline = time.time() + 20
         current = ac.get(job["id"])
+        while time.time() < deadline and "upstream unavailable" not in (current["error"] or ""):
+            time.sleep(0.2)
+            current = ac.get(job["id"])
         assert current["status"] == "running" and "upstream unavailable" in (current["error"] or "")
         ac.action(job["id"], "cancel")
         assert wait_for(ac, job["id"], {"cancelled"})["status"] == "cancelled"

pywget-2.0.4/pywget/config.py

@@ -1,93 +0,0 @@
-"""Configuration helpers: config dir, API token, server configuration."""
-
-from __future__ import annotations
-
-import os
-import secrets
-import sys
-from dataclasses import dataclass, field
-from pathlib import Path
-from typing import Any, List, Optional
-
-DEFAULT_HOST = "127.0.0.1"
-DEFAULT_PORT = 8765
-DEFAULT_AGENT_PORT = 8766
-LOOPBACK_HOSTS = ("127.0.0.1", "localhost", "::1")
-
-
-def config_dir() -> Path:
-    override = os.environ.get("PYWGET_CONFIG_DIR")
-    if override:
-        return Path(override).expanduser()
-    if sys.platform == "win32":
-        base = os.environ.get("APPDATA") or str(Path.home() / "AppData" / "Roaming")
-        return Path(base) / "pywget"
-    base = os.environ.get("XDG_CONFIG_HOME") or str(Path.home() / ".config")
-    return Path(base) / "pywget"
-
-
-def token_path() -> Path:
-    return config_dir() / "token"
-
-
-def load_token(explicit: Optional[str] = None, create: bool = False) -> Optional[str]:
-    """Resolve the API token: explicit > $PYWGET_TOKEN > token file (> generate)."""
-    if explicit:
-        return explicit.strip()
-    env = os.environ.get("PYWGET_TOKEN")
-    if env:
-        return env.strip()
-    path = token_path()
-    try:
-        text = path.read_text(encoding="utf-8").strip()
-        if text:
-            return text
-    except OSError:
-        pass
-    if not create:
-        return None
-    token = secrets.token_urlsafe(32)
-    path.parent.mkdir(parents=True, exist_ok=True)
-    try:
-        fd = os.open(str(path), os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
-    except FileExistsError:  # lost a race with another process
-        return path.read_text(encoding="utf-8").strip()
-    with os.fdopen(fd, "w", encoding="utf-8") as handle:
-        handle.write(token + "\n")
-    return token
-
-
-@dataclass
-class ServerConfig:
-    host: str = DEFAULT_HOST
-    port: int = DEFAULT_PORT
-    token: str = ""
-    root: Path = field(default_factory=lambda: Path.home() / "Downloads" / "pywget")
-    state_dir: Path = field(default_factory=lambda: config_dir() / "server")
-    max_jobs: int = 3
-    connections: int = 4
-    retries: int = 5
-    limit_rate: Any = 0
-    deny_private: bool = False
-    allowed_hosts: List[str] = field(default_factory=list)
-    tls_cert: Optional[str] = None
-    tls_key: Optional[str] = None
-    ui: bool = True
-    upstream: Optional[str] = None
-    upstream_token: Optional[str] = None
-    upstream_verify: bool = True
-    upstream_ca: Optional[str] = None
-
-    def __post_init__(self) -> None:
-        self.root = Path(self.root).expanduser()
-        self.state_dir = Path(self.state_dir).expanduser()
-        if not self.token:
-            raise ValueError("a non-empty API token is required")
-        if not 1 <= self.max_jobs <= 256:
-            raise ValueError("max_jobs must be between 1 and 256")
-        if bool(self.tls_cert) != bool(self.tls_key):
-            raise ValueError("--tls-cert and --tls-key must be given together")
-
-    @property
-    def is_agent(self) -> bool:
-        return bool(self.upstream)