booru-dl 2.1.0__tar.gz

booru_dl-2.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 wexqk-boo
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

booru_dl-2.1.0/PKG-INFO

@@ -0,0 +1,216 @@
+Metadata-Version: 2.4
+Name: booru-dl
+Version: 2.1.0
+Summary: Async batch downloader and sync tool for booru boards
+License: MIT
+Project-URL: Homepage, https://github.com/wexqk-boo/Booru-Downloader-py
+Project-URL: Issues, https://github.com/wexqk-boo/Booru-Downloader-py/issues
+Keywords: booru,downloader,gelbooru,rule34,danbooru,asyncio,cli
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: End Users/Desktop
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Internet :: WWW/HTTP
+Classifier: Topic :: Multimedia :: Graphics
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: aiohttp<4,>=3.10
+Requires-Dist: aiofiles>=24.1
+Requires-Dist: aiosqlite>=0.20
+Requires-Dist: imagehash>=4.3
+Requires-Dist: pillow>=10
+Requires-Dist: rapidfuzz>=3
+Requires-Dist: rich>=13
+Requires-Dist: questionary>=2.0
+Requires-Dist: uvloop>=0.21; sys_platform != "win32"
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == "dev"
+Requires-Dist: ruff>=0.6; extra == "dev"
+Dynamic: license-file
+
+# bdl
+
+Async batch downloader and sync tool for booru boards (Gelbooru, Rule34,
+Xbooru, Safebooru, Danbooru, Konachan, Yande.re, Tbib, e621, e926).
+
+## Features
+
+- **Async streaming downloads** — `asyncio` + `aiohttp`, files streamed to disk
+  in chunks, concurrency bounded by `asyncio.Semaphore`.
+- **Four-level deduplication** — `uid` in SQLite, the file already present on
+  disk, an optional sha256 check, and a perceptual pHash so the same artwork
+  reposted to another board under a new id is skipped too. The first three
+  levels are free; the pHash has to see the bytes, so a repost is downloaded,
+  recognised and then deleted instead of being kept.
+- **Aspect-ratio filtering** — `--wallpapers`, `--avatars`, or a custom
+  `--aspect-ratio MIN:MAX`, applied after the API response so nothing extra is
+  downloaded. Boards that omit image dimensions are kept by default; pass
+  `--aspect-unknown skip` to drop them instead.
+- **Tag-family blacklist** — a root tag pulls in its related sub-tags from
+  `~/.config/bdl/blacklist.txt`, plus a per-run `--blacklist` override.
+- **DNS-over-HTTPS and proxy support** — a built-in `DoHResolver` (Cloudflare)
+  works around local DNS blocks on image CDNs, and an upstream proxy can be set.
+- **uvloop when available** — the event loop switches to uvloop on every
+  platform but Windows, which speeds up a batch of many small transfers.
+- **Typo tolerance** — a tag that is not in the dictionary is matched against
+  the popular tags and corrected before the query goes out (`mku` → `miku`).
+- **Image previews** — inside Kitty, finished pictures are drawn once the batch
+  ends as a compact gallery of thumbnail rows rather than a column running off
+  the screen; `--no-preview` turns them off.
+- **Interactive wizard** — `bdl` on a terminal (or `bdl tui`) walks through the
+  download: boards with checkboxes, tags with live tag suggestions, limit, sort
+  order, rating, geometry and an optional list of tags to exclude, then prints a
+  summary before starting. The interface ships in Russian (default) and English;
+  `questionary` supplies the arrow-key menus and `rich` the section headers.
+- **Machine-readable output** — `--json` on `search` and `status`.
+
+## Installation
+
+Requires Python 3.11+. The repository is **Booru-Downloader-py**; the console
+command it installs is **`bdl`**.
+
+```bash
+git clone https://github.com/wexqk-boo/Booru-Downloader-py.git
+cd Booru-Downloader-py
+python3 -m venv .venv
+./.venv/bin/pip install -e .          # or: pip install -e '.[dev]' for tests/ruff
+```
+
+The package installs a single entry point: `bdl`.
+
+To put it on `PATH`:
+
+```bash
+./install.sh          # symlink the entry point into ~/.local/bin
+./install.sh --copy   # copy it instead
+./install.sh --remove # uninstall
+```
+
+Run `bdl` on a terminal to open the wizard. Without a terminal it prints usage
+and exits `2`, so scripts never block on a prompt. If an old install left a
+fish function named `bdl`, it shadows the binary — remove it
+(`rm ~/.config/fish/functions/bdl.fish`) and run `functions -e bdl` once;
+`./install.sh --remove` cleans that up together with the stale helper.
+
+`rapidfuzz` powers tag suggestions and typo correction; without it the code
+falls back to the standard-library `difflib`. `imagehash` and `pillow` enable
+perceptual dedup, and `uvloop` speeds up the event loop everywhere but Windows.
+
+## Usage
+
+```bash
+# Download every source, tag "miku", wallpaper aspect ratio only
+bdl dl -s all -t "miku" --wallpapers
+
+# Preview results without downloading
+bdl search -s gelbooru -t "school_uniform" --limit 20
+
+# Download statistics from the SQLite state
+bdl status
+
+# Daemon-style sync: repeated passes, already-downloaded posts skipped
+bdl sync -s all -t "rating:general" --interval 900
+
+# Blacklist management
+bdl blacklist list
+bdl blacklist add loli
+```
+
+Other subcommands: `tui` (interactive wizard), `prune-stale` (forget DB rows
+whose files are gone), `config` (`get` / `set` / `show`), and `tags` (tag
+suggestions).
+
+`prune-stale` only inspects paths under the current output root, so a run
+against a temporary `--out` cannot forget the real library. Use `--all` to
+check every recorded path, or `--out DIR` to pick a different scope.
+
+### Wizard
+
+```bash
+bdl tui     # or bare `bdl` on a terminal, which reaches the same wizard
+```
+
+Rating choices are Safe / Suggestive / NSFW / All (`--safe`, `--suggestive`,
+`--nsfw` on the command line), and geometry offers Wallpapers, Avatars, a
+custom `MIN:MAX` range, or no filter. `MIN:MAX` accepts `1.5:1.8`, `1,5:1,8`,
+`1.5/1.8` and width:height shorthands such as `16:9` or `3:4`.
+
+The whole wizard is five screens and takes well under a minute: boards, the tag
+query, limit and geometry, an optional line of tags to exclude, then the
+summary. It never asks about concurrency — 8 parallel transfers keep a 100 Mbit
+line busy — and the JSON/sha256/re-download switches stay off unless you pass
+them to `dl` yourself.
+
+The wizard asks for the output folder only until one is known: the first answer
+is saved to the config and later runs reuse it silently, unless `BDL_OUT` is
+set, which wins outright and is never written back. The exclusion step is a
+single comma-separated field — a root tag such as `ai` pulls in its whole family,
+an empty answer excludes nothing. Tag fields complete from the local Danbooru
+database as you type, and the run ends with a summary before the final
+confirmation.
+
+Common flags for `dl` / `sync`: `--limit`, `--per-page`, `--out`,
+`--concurrency`, `--sidecar`, `--checksum`, `--force`, `--min-score`,
+`--rating-include`, `--max-size-mb`, `--blacklist`, `--no-progress`,
+`--no-preview`, `--no-notify`, `--timeout`.
+
+## Directory layout
+
+Files are written as `<out_root>/<source>/<bucket>/[videos/]/<filename>`, with
+all sidecar JSON collected in a `json/` folder:
+
+```
+<out_root>/<source>/
+├── safe/            # ratings: safe, s, general, g
+│   ├── <file>.jpg
+│   ├── videos/      # mp4, webm, mov, avi, mkv, flv, m4v, wmv
+│   └── json/        # <file>.jpg.json sidecars (with --sidecar)
+├── suggestive/      # ratings: questionable, q, sensitive
+└── nsfw/            # ratings: explicit, e
+```
+
+The default `out_root` comes from `BDL_OUT`, then the `out` key in the config
+file, then `~/Pictures/booru`. Unknown or missing ratings land in `nsfw/`.
+
+## Configuration
+
+- Config file: `~/.config/bdl/config.json`
+- Download state (SQLite): `~/.local/share/bdl/downloads.sqlite3`
+- Tag blacklist: `~/.config/bdl/blacklist.txt`
+- Tag database cache: `~/.cache/bdl/`
+- Log: `~/.local/state/bdl/bdl.log`
+
+Environment variables:
+
+- `BDL_OUT` — override the output root (`BOORU_DL_OUT` still works).
+- `BDL_LOG_FILE` — duplicate logs to a file (`BOORU_DL_LOG_FILE` still works).
+- `BDL_LANG` — wizard language, `ru` (default) or `en`. Also settable through
+  `bdl config set lang en`.
+- `PROXY_URL` — route requests through a proxy.
+
+### Board blocks
+
+Xbooru's DAPI needs no API key, but Cloudflare fronts the site and sometimes
+challenges an IP with HTTP 403. Requests to Xbooru therefore carry a full set of
+browser headers. If the IP is still challenged, the affected files are counted
+as `blocked` instead of filling the screen with warnings, and the reason is
+reported once in the batch summary at the end. Setting `PROXY_URL` is the usual
+way out.
+- `bdl config set out ~/Pictures/booru` — default output root for every command
+  that takes `--out` (`BDL_OUT` still wins over it).
+
+## Development
+
+```bash
+./.venv/bin/python -m pytest tests -q    # unit tests, no network
+./.venv/bin/ruff check bdl tests
+```
+
+## License
+
+MIT

booru_dl-2.1.0/README.md

@@ -0,0 +1,181 @@
+# bdl
+
+Async batch downloader and sync tool for booru boards (Gelbooru, Rule34,
+Xbooru, Safebooru, Danbooru, Konachan, Yande.re, Tbib, e621, e926).
+
+## Features
+
+- **Async streaming downloads** — `asyncio` + `aiohttp`, files streamed to disk
+  in chunks, concurrency bounded by `asyncio.Semaphore`.
+- **Four-level deduplication** — `uid` in SQLite, the file already present on
+  disk, an optional sha256 check, and a perceptual pHash so the same artwork
+  reposted to another board under a new id is skipped too. The first three
+  levels are free; the pHash has to see the bytes, so a repost is downloaded,
+  recognised and then deleted instead of being kept.
+- **Aspect-ratio filtering** — `--wallpapers`, `--avatars`, or a custom
+  `--aspect-ratio MIN:MAX`, applied after the API response so nothing extra is
+  downloaded. Boards that omit image dimensions are kept by default; pass
+  `--aspect-unknown skip` to drop them instead.
+- **Tag-family blacklist** — a root tag pulls in its related sub-tags from
+  `~/.config/bdl/blacklist.txt`, plus a per-run `--blacklist` override.
+- **DNS-over-HTTPS and proxy support** — a built-in `DoHResolver` (Cloudflare)
+  works around local DNS blocks on image CDNs, and an upstream proxy can be set.
+- **uvloop when available** — the event loop switches to uvloop on every
+  platform but Windows, which speeds up a batch of many small transfers.
+- **Typo tolerance** — a tag that is not in the dictionary is matched against
+  the popular tags and corrected before the query goes out (`mku` → `miku`).
+- **Image previews** — inside Kitty, finished pictures are drawn once the batch
+  ends as a compact gallery of thumbnail rows rather than a column running off
+  the screen; `--no-preview` turns them off.
+- **Interactive wizard** — `bdl` on a terminal (or `bdl tui`) walks through the
+  download: boards with checkboxes, tags with live tag suggestions, limit, sort
+  order, rating, geometry and an optional list of tags to exclude, then prints a
+  summary before starting. The interface ships in Russian (default) and English;
+  `questionary` supplies the arrow-key menus and `rich` the section headers.
+- **Machine-readable output** — `--json` on `search` and `status`.
+
+## Installation
+
+Requires Python 3.11+. The repository is **Booru-Downloader-py**; the console
+command it installs is **`bdl`**.
+
+```bash
+git clone https://github.com/wexqk-boo/Booru-Downloader-py.git
+cd Booru-Downloader-py
+python3 -m venv .venv
+./.venv/bin/pip install -e .          # or: pip install -e '.[dev]' for tests/ruff
+```
+
+The package installs a single entry point: `bdl`.
+
+To put it on `PATH`:
+
+```bash
+./install.sh          # symlink the entry point into ~/.local/bin
+./install.sh --copy   # copy it instead
+./install.sh --remove # uninstall
+```
+
+Run `bdl` on a terminal to open the wizard. Without a terminal it prints usage
+and exits `2`, so scripts never block on a prompt. If an old install left a
+fish function named `bdl`, it shadows the binary — remove it
+(`rm ~/.config/fish/functions/bdl.fish`) and run `functions -e bdl` once;
+`./install.sh --remove` cleans that up together with the stale helper.
+
+`rapidfuzz` powers tag suggestions and typo correction; without it the code
+falls back to the standard-library `difflib`. `imagehash` and `pillow` enable
+perceptual dedup, and `uvloop` speeds up the event loop everywhere but Windows.
+
+## Usage
+
+```bash
+# Download every source, tag "miku", wallpaper aspect ratio only
+bdl dl -s all -t "miku" --wallpapers
+
+# Preview results without downloading
+bdl search -s gelbooru -t "school_uniform" --limit 20
+
+# Download statistics from the SQLite state
+bdl status
+
+# Daemon-style sync: repeated passes, already-downloaded posts skipped
+bdl sync -s all -t "rating:general" --interval 900
+
+# Blacklist management
+bdl blacklist list
+bdl blacklist add loli
+```
+
+Other subcommands: `tui` (interactive wizard), `prune-stale` (forget DB rows
+whose files are gone), `config` (`get` / `set` / `show`), and `tags` (tag
+suggestions).
+
+`prune-stale` only inspects paths under the current output root, so a run
+against a temporary `--out` cannot forget the real library. Use `--all` to
+check every recorded path, or `--out DIR` to pick a different scope.
+
+### Wizard
+
+```bash
+bdl tui     # or bare `bdl` on a terminal, which reaches the same wizard
+```
+
+Rating choices are Safe / Suggestive / NSFW / All (`--safe`, `--suggestive`,
+`--nsfw` on the command line), and geometry offers Wallpapers, Avatars, a
+custom `MIN:MAX` range, or no filter. `MIN:MAX` accepts `1.5:1.8`, `1,5:1,8`,
+`1.5/1.8` and width:height shorthands such as `16:9` or `3:4`.
+
+The whole wizard is five screens and takes well under a minute: boards, the tag
+query, limit and geometry, an optional line of tags to exclude, then the
+summary. It never asks about concurrency — 8 parallel transfers keep a 100 Mbit
+line busy — and the JSON/sha256/re-download switches stay off unless you pass
+them to `dl` yourself.
+
+The wizard asks for the output folder only until one is known: the first answer
+is saved to the config and later runs reuse it silently, unless `BDL_OUT` is
+set, which wins outright and is never written back. The exclusion step is a
+single comma-separated field — a root tag such as `ai` pulls in its whole family,
+an empty answer excludes nothing. Tag fields complete from the local Danbooru
+database as you type, and the run ends with a summary before the final
+confirmation.
+
+Common flags for `dl` / `sync`: `--limit`, `--per-page`, `--out`,
+`--concurrency`, `--sidecar`, `--checksum`, `--force`, `--min-score`,
+`--rating-include`, `--max-size-mb`, `--blacklist`, `--no-progress`,
+`--no-preview`, `--no-notify`, `--timeout`.
+
+## Directory layout
+
+Files are written as `<out_root>/<source>/<bucket>/[videos/]/<filename>`, with
+all sidecar JSON collected in a `json/` folder:
+
+```
+<out_root>/<source>/
+├── safe/            # ratings: safe, s, general, g
+│   ├── <file>.jpg
+│   ├── videos/      # mp4, webm, mov, avi, mkv, flv, m4v, wmv
+│   └── json/        # <file>.jpg.json sidecars (with --sidecar)
+├── suggestive/      # ratings: questionable, q, sensitive
+└── nsfw/            # ratings: explicit, e
+```
+
+The default `out_root` comes from `BDL_OUT`, then the `out` key in the config
+file, then `~/Pictures/booru`. Unknown or missing ratings land in `nsfw/`.
+
+## Configuration
+
+- Config file: `~/.config/bdl/config.json`
+- Download state (SQLite): `~/.local/share/bdl/downloads.sqlite3`
+- Tag blacklist: `~/.config/bdl/blacklist.txt`
+- Tag database cache: `~/.cache/bdl/`
+- Log: `~/.local/state/bdl/bdl.log`
+
+Environment variables:
+
+- `BDL_OUT` — override the output root (`BOORU_DL_OUT` still works).
+- `BDL_LOG_FILE` — duplicate logs to a file (`BOORU_DL_LOG_FILE` still works).
+- `BDL_LANG` — wizard language, `ru` (default) or `en`. Also settable through
+  `bdl config set lang en`.
+- `PROXY_URL` — route requests through a proxy.
+
+### Board blocks
+
+Xbooru's DAPI needs no API key, but Cloudflare fronts the site and sometimes
+challenges an IP with HTTP 403. Requests to Xbooru therefore carry a full set of
+browser headers. If the IP is still challenged, the affected files are counted
+as `blocked` instead of filling the screen with warnings, and the reason is
+reported once in the batch summary at the end. Setting `PROXY_URL` is the usual
+way out.
+- `bdl config set out ~/Pictures/booru` — default output root for every command
+  that takes `--out` (`BDL_OUT` still wins over it).
+
+## Development
+
+```bash
+./.venv/bin/python -m pytest tests -q    # unit tests, no network
+./.venv/bin/ruff check bdl tests
+```
+
+## License
+
+MIT

booru_dl-2.1.0/bdl/__init__.py

@@ -0,0 +1,27 @@
+"""Standalone CLI for batch downloading and syncing content from Booru boards.
+
+Self-contained: own providers, DoH resolver, models, dedup database and tag
+dictionaries. Runs as ``python -m bdl`` or via the ``bdl`` console script.
+"""
+
+__version__ = "2.1.0"
+
+#: Repository the project is published from.
+REPO_SLUG = "github.com/wexqk-boo/Booru-Downloader-py"
+
+#: User-Agent for normal outgoing API and media requests.
+USER_AGENT = f"bdl/{__version__} ({REPO_SLUG})"
+
+#: Xbooru has an open DAPI with no keys, but its Cloudflare edge challenges a
+#: client that announces itself as a script, so both the API call and the media
+#: fetch present a full set of browser headers instead.
+XBOORU_USER_AGENT = "Mozilla/5.0 (X11; Linux x86_64; rv:132.0) Gecko/20100101 Firefox/132.0"
+
+XBOORU_HEADERS: dict[str, str] = {
+    "User-Agent": XBOORU_USER_AGENT,
+    "Referer": "https://xbooru.com/index.php?page=post&s=list",
+    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
+    "Accept-Language": "en-US,en;q=0.5",
+}
+
+__all__ = ["REPO_SLUG", "USER_AGENT", "XBOORU_HEADERS", "XBOORU_USER_AGENT", "__version__"]

booru_dl-2.1.0/bdl/__main__.py

@@ -0,0 +1,8 @@
+"""Entry point: python -m bdl ..."""
+
+import sys
+
+from .cli import main
+
+if __name__ == "__main__":
+    sys.exit(main())

booru_dl-2.1.0/bdl/blacklist.py

@@ -0,0 +1,74 @@
+"""Tag blacklist: persistent file and management helpers.
+
+Default file: ``~/.config/bdl/blacklist.txt`` (one tag per line, ``#`` starts a
+comment). Posts matching any of these tags are skipped during download, and a
+base tag such as ``ai`` also pulls in its whole family.
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+from .config import xdg_config_home
+from .tags.families import expand_blacklist_tags
+
+
+def default_blacklist_path() -> Path:
+    return xdg_config_home() / "bdl" / "blacklist.txt"
+
+
+def load_blacklist(path: Path | str | None = None) -> set[str]:
+    """Read the tag list in lowercase; a missing file yields an empty set."""
+    file_path = Path(path).expanduser() if path else default_blacklist_path()
+    if not file_path.is_file():
+        return set()
+    tags: set[str] = set()
+    for line in file_path.read_text(encoding="utf-8").splitlines():
+        tag = line.strip().lower()
+        if tag and not tag.startswith("#"):
+            tags.add(tag)
+    return tags
+
+
+def save_blacklist(tags: set[str], path: Path | str | None = None) -> Path:
+    """Write the sorted tag list and return the file path."""
+    file_path = Path(path).expanduser() if path else default_blacklist_path()
+    file_path.parent.mkdir(parents=True, exist_ok=True)
+    lines = [f"{tag}\n" for tag in sorted(tags)]
+    file_path.write_text("".join(lines), encoding="utf-8")
+    return file_path
+
+
+def expand_tag(tag: str) -> set[str]:
+    """Tag family: the tag itself plus related sub-tags (ai/furry/guro/...)."""
+    clean = tag.strip().lower()
+    if not clean:
+        return set()
+    expanded = set(expand_blacklist_tags(clean))
+    expanded.add(clean)
+    return expanded
+
+
+def add_tag(tag: str, path: Path | str | None = None) -> int:
+    """Add a tag and its family; return how many entries were added."""
+    expanded = expand_tag(tag)
+    if not expanded:
+        return 0
+    tags = load_blacklist(path)
+    before = len(tags)
+    tags |= expanded
+    if len(tags) != before:
+        save_blacklist(tags, path)
+    return len(tags) - before
+
+
+def remove_tag(tag: str, path: Path | str | None = None) -> int:
+    """Remove a tag and its family if present; return how many entries were removed."""
+    clean = tag.strip().lower()
+    tags = load_blacklist(path)
+    to_remove = expand_tag(clean) & tags
+    if not to_remove:
+        return 0
+    tags -= to_remove
+    save_blacklist(tags, path)
+    return len(to_remove)