library 3.2.7__tar.gz → 3.2.11__tar.gz

{library-3.2.7 → library-3.2.11}/.github/README.md

@@ -10,7 +10,9 @@ tracking online video playlists (eg. YouTube subscriptions) and scheduling brows
 
 [![Downloads](https://static.pepy.tech/badge/library)](https://pepy.tech/project/library)
 
-## Install
+## Getting started
+
+<details><summary>Install</summary>
 
 Linux recommended but [Windows setup instructions](./Windows.md) available.
 
@@ -24,7 +26,31 @@ Required: `ffmpeg`
 
 Some features work better with: `mpv`, `fd-find`, `fish`
 
-## Getting started
+Optional Python extras are grouped under `library[deluxe]`. This includes
+`gallery-dl` (used by `gallery-add`), `internetarchive` (used for Internet
+Archive downloads), and `cdx_toolkit` (used by the `--webcache`/`--wayback`
+CDX extractor):
+
+    pip install "library[deluxe]"
+
+Torrent metadata commands (`torrents-add` and `torrents-dump`) require the
+`libtorrent` Python bindings. They are included by `library[deluxe]` when using
+Python 3.12 or 3.13; Python 3.14 wheels are not available yet, so use Python
+3.13 for torrent support.
+
+If the PyPI wheel is unavailable for your Python or platform, these system
+packages provide the bindings:
+
+* Debian/Ubuntu: `sudo apt install python3-libtorrent`
+* Fedora: `sudo dnf install rb_libtorrent-python3`
+* macOS/Homebrew: `brew install libtorrent-rasterbar`
+
+The Debian, Fedora, and Homebrew packages are built for their system or
+Homebrew Python and may not be importable from a separate uv virtualenv. There
+is no official Scoop package for the Python bindings; on Windows, use Python
+3.13 and `pip install "library[deluxe]"`.
+
+</details>
 
 <details><summary>Local media</summary>
 
@@ -99,7 +125,7 @@ To stop playing press Ctrl+C in either the terminal or mpv
 <details><summary>List all subcommands</summary>
 
     $ library
-    library (v3.2.007; 102 subcommands)
+    library (v3.2.011; 102 subcommands)
 
     Create database subcommands:
     ╭─────────────────┬──────────────────────────────────────────╮
@@ -797,6 +823,14 @@ BTW, for some cols like time_deleted you'll need to specify a where clause so th
         --page-key start --page-start 0 --page-step 50 --fixed-pages 14 --stop-pages-no-match 1 \
         plab.db https://plab/forum/tracker.php?o=(string replace ' ' \n -- 1 4 7 10 15)&s=2&tm=-1&f=(string replace ' ' \n -- 1670 1768 60 1671 1644 1672 1111 508 555 1112 1718 1143 1717 1851 1713 1712 1775 1674 902 1675 36 1830 1803 1831 1741 1676 1677 1780 1110 1124 1784 1769 1793 1797 1804 1819 1825 1836 1842 1846 1857 1861 1867 1451 1788 1789 1792 1798 1805 1820 1826 1837 1843 1847 1856 1862 1868 284 1853 1823 1800 1801 1719 997 1818 1849 1711 1791 1762)
 
+    Recursively add every link on a site (glob excludes prune subtrees from the crawl)
+
+        library links-add links.db --recursive https://example.com/docs/ --path-exclude '*/tags/*'
+
+    Add every archived link for a site from the Internet Archive CDX index
+
+        library links-add links.db --webcache https://example.com/docs/
+
 
 </details>
 
@@ -1130,6 +1164,31 @@ BTW, for some cols like time_deleted you'll need to specify a where clause so th
         library links https://en.wikipedia.org/wiki/List_of_bacon_dishes --path-include https://en.wikipedia.org/wiki/ --after-include famous
         https://en.wikipedia.org/wiki/Omelette
 
+    Download the filtered links. Internet Archive details pages use the Internet Archive downloader and preserve all original files:
+
+        library extract-links --download https://readrare.com/
+
+    Mirror a site recursively. Only links within the same domain and parent directory as the given
+    path are followed; links pointing elsewhere are downloaded but not followed. Includes/excludes
+    accept globs, so you can prune whole subtrees while crawling:
+
+        library links --recursive --download https://example.com/docs/
+        library links -R --download https://example.com/docs/ --path-exclude '*/tag/*' '*/tags/*'
+
+    Extract links from a web archive CDX index instead of crawling live pages. Internet Archive
+    Wayback snapshots are emitted by default. Wayback URLs are collapsed to their original
+    archived URL while crawling: sibling pages that only exist under a different timestamp are
+    still followed, but multiple captures of the same page are collapsed so only one copy of
+    each page is downloaded. Use --webcache-from/--webcache-to to target a time period:
+
+        library links --webcache https://example.com/docs/
+        library links --webcache --recursive --download https://example.com/docs/
+        library links --webcache --webcache-source cc https://example.com/docs/
+        library links --webcache --webcache-from 2020 --webcache-to 2021 https://example.com/docs/
+
+    Common tracking and analytics URLs, including Google Tag Manager and Google Analytics,
+    are excluded by default. Use --path-exclude to replace the default exclusions when needed.
+
     Read from local clipboard and filter out links based on nearby plain text
 
         library links --local-html (cb -t text/html | psub) --after-exclude paranormal spooky horror podcast tech fantasy supernatural lecture sport
@@ -2320,7 +2379,7 @@ Inspired somewhat by https://nikkhokkho.sourceforge.io/?page=FileOptimizer
         | /home/xk/github/xk/lb/.ruff_cache/    |   19.5 kB |     100 |
         | /home/xk/github/xk/lb/.gitattributes  | 119 Bytes |         |
         | /home/xk/github/xk/lb/.mypy_cache/    | 280 Bytes |       4 |
-        | /home/xk/github/xk/lb/.pdm-python     |  15 Bytes |         |
+        | /home/xk/github/xk/lb/.venv            |  15 Bytes |         |
 
     Only include files with a specific depth
 
@@ -2493,6 +2552,17 @@ Inspired somewhat by https://nikkhokkho.sourceforge.io/?page=FileOptimizer
         library download dl.db --include https://www.youtube.com/watch?v=YE7VzlLtp-4
         library download dl.db -s https://www.youtube.com/watch?v=YE7VzlLtp-4  # equivalent
 
+    Recursively download a site (filesystem profile). Only links within the same domain and
+    parent directory as the seed are followed; other links are downloaded but not followed.
+
+        library download links.db --filesystem --recursive --prefix ~/mirror/ https://example.com/docs/
+
+    Download a site's archived pages from the Internet Archive Wayback Machine using the CDX API.
+    `--webcache` implies `--links`; add `--recursive` to also parse the archived pages for
+    further links:
+
+        library download links.db --filesystem --webcache --prefix ~/mirror/ https://example.com/docs/
+
     Maximizing the variety of subdomains
 
         library download photos.db --photos --image --sort "ROW_NUMBER() OVER ( PARTITION BY SUBSTR(m.path, INSTR(m.path, '//') + 2, INSTR( SUBSTR(m.path, INSTR(m.path, '//') + 2), '/') - 1) )"

{library-3.2.7 → library-3.2.11}/PKG-INFO

@@ -1,8 +1,9 @@
-Metadata-Version: 2.1
+Metadata-Version: 2.3
 Name: library
-Version: 3.2.7
+Version: 3.2.11
 Summary: xk media library
-Author-Email: Jacob Chapman <7908073+chapmanjacobd@users.noreply.github.com>
+Author: Jacob Chapman
+Author-email: Jacob Chapman <7908073+chapmanjacobd@users.noreply.github.com>
 License: BSD 3-Clause License
          
          Copyright (c) 2024, Jacob Chapman
@@ -31,16 +32,10 @@ License: BSD 3-Clause License
          CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
          OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
          OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-         
-Project-URL: documentation, https://github.com/chapmanjacobd/library#usage
-Project-URL: homepage, https://github.com/chapmanjacobd/library#readme
-Project-URL: repository, https://github.com/chapmanjacobd/library/
-Requires-Python: >=3.12
 Requires-Dist: beautifulsoup4>=4.10.0
 Requires-Dist: sqlite-utils>=3.38
 Requires-Dist: ipython>=9.0.0
 Requires-Dist: ffmpeg-python
-Requires-Dist: gallery-dl
 Requires-Dist: yt-dlp[default]
 Requires-Dist: natsort
 Requires-Dist: charset-normalizer
@@ -61,49 +56,56 @@ Requires-Dist: screeninfo
 Requires-Dist: tabulate
 Requires-Dist: tinytag
 Requires-Dist: tldextract
-Requires-Dist: pyreadline3; platform_system == "Windows"
-Provides-Extra: dev
-Requires-Dist: black; extra == "dev"
-Requires-Dist: isort; extra == "dev"
-Requires-Dist: ssort; extra == "dev"
-Provides-Extra: test
-Requires-Dist: ruff; extra == "test"
-Requires-Dist: pytest; extra == "test"
-Requires-Dist: pytest-regressions; extra == "test"
-Requires-Dist: freezegun; extra == "test"
-Requires-Dist: pandas; extra == "test"
-Requires-Dist: pyfakefs; extra == "test"
+Requires-Dist: pyreadline3 ; sys_platform == 'win32'
+Requires-Dist: aiohttp ; extra == 'deluxe'
+Requires-Dist: annoy ; extra == 'deluxe'
+Requires-Dist: catt ; extra == 'deluxe'
+Requires-Dist: cdx-toolkit ; extra == 'deluxe'
+Requires-Dist: curl-cffi ; extra == 'deluxe'
+Requires-Dist: gallery-dl ; extra == 'deluxe'
+Requires-Dist: geopandas ; extra == 'deluxe'
+Requires-Dist: img2pdf ; extra == 'deluxe'
+Requires-Dist: internetarchive ; extra == 'deluxe'
+Requires-Dist: ocrmypdf ; extra == 'deluxe'
+Requires-Dist: openpyxl ; extra == 'deluxe'
+Requires-Dist: paramiko ; extra == 'deluxe'
+Requires-Dist: pdf2image ; extra == 'deluxe'
+Requires-Dist: pillow ; extra == 'deluxe'
+Requires-Dist: pyexiftool ; extra == 'deluxe'
+Requires-Dist: pymcdm ; extra == 'deluxe'
+Requires-Dist: pyvirtualdisplay ; extra == 'deluxe'
+Requires-Dist: qbittorrent-api ; extra == 'deluxe'
+Requires-Dist: scikit-learn ; extra == 'deluxe'
+Requires-Dist: selenium ; extra == 'deluxe'
+Requires-Dist: libtorrent ; python_full_version < '3.14' and extra == 'deluxe'
+Requires-Dist: tqdm ; extra == 'deluxe'
+Requires-Dist: wordllama>=0.2.7.post0 ; extra == 'deluxe'
+Requires-Dist: xattr ; sys_platform != 'win32' and extra == 'deluxe'
+Requires-Dist: black ; extra == 'dev'
+Requires-Dist: isort ; extra == 'dev'
+Requires-Dist: ssort ; extra == 'dev'
+Requires-Dist: brotab ; extra == 'fat'
+Requires-Dist: camelot-py>=0.11.0 ; extra == 'fat'
+Requires-Dist: ghostscript ; extra == 'fat'
+Requires-Dist: opencv-python ; extra == 'fat'
+Requires-Dist: orjson ; extra == 'fat'
+Requires-Dist: selenium-wire-lw ; extra == 'fat'
+Requires-Dist: subliminal ; extra == 'fat'
+Requires-Dist: textract-py3==2.0.1 ; extra == 'fat'
+Requires-Dist: ruff ; extra == 'test'
+Requires-Dist: pytest ; extra == 'test'
+Requires-Dist: pytest-regressions ; extra == 'test'
+Requires-Dist: freezegun ; extra == 'test'
+Requires-Dist: pandas ; extra == 'test'
+Requires-Dist: pyfakefs ; extra == 'test'
+Requires-Python: >=3.12
+Project-URL: documentation, https://github.com/chapmanjacobd/library#usage
+Project-URL: homepage, https://github.com/chapmanjacobd/library#readme
+Project-URL: repository, https://github.com/chapmanjacobd/library/
 Provides-Extra: deluxe
-Requires-Dist: aiohttp; extra == "deluxe"
-Requires-Dist: annoy; extra == "deluxe"
-Requires-Dist: catt; extra == "deluxe"
-Requires-Dist: curl-cffi; extra == "deluxe"
-Requires-Dist: geopandas; extra == "deluxe"
-Requires-Dist: img2pdf; extra == "deluxe"
-Requires-Dist: ocrmypdf; extra == "deluxe"
-Requires-Dist: openpyxl; extra == "deluxe"
-Requires-Dist: paramiko; extra == "deluxe"
-Requires-Dist: pdf2image; extra == "deluxe"
-Requires-Dist: pillow; extra == "deluxe"
-Requires-Dist: PyExifTool; extra == "deluxe"
-Requires-Dist: pymcdm; extra == "deluxe"
-Requires-Dist: pyvirtualdisplay; extra == "deluxe"
-Requires-Dist: qbittorrent-api; extra == "deluxe"
-Requires-Dist: scikit-learn; extra == "deluxe"
-Requires-Dist: selenium; extra == "deluxe"
-Requires-Dist: libtorrent; extra == "deluxe"
-Requires-Dist: tqdm; extra == "deluxe"
-Requires-Dist: wordllama>=0.2.7.post0; extra == "deluxe"
-Requires-Dist: xattr; extra == "deluxe"
+Provides-Extra: dev
 Provides-Extra: fat
-Requires-Dist: brotab; extra == "fat"
-Requires-Dist: camelot-py>=0.11.0; extra == "fat"
-Requires-Dist: ghostscript; extra == "fat"
-Requires-Dist: opencv-python; extra == "fat"
-Requires-Dist: orjson; extra == "fat"
-Requires-Dist: selenium-wire-lw; extra == "fat"
-Requires-Dist: subliminal; extra == "fat"
-Requires-Dist: textract-py3==2.0.1; extra == "fat"
+Provides-Extra: test
 Description-Content-Type: text/markdown
 
 # library (media toolkit)
@@ -118,7 +120,9 @@ tracking online video playlists (eg. YouTube subscriptions) and scheduling brows
 
 [![Downloads](https://static.pepy.tech/badge/library)](https://pepy.tech/project/library)
 
-## Install
+## Getting started
+
+<details><summary>Install</summary>
 
 Linux recommended but [Windows setup instructions](./Windows.md) available.
 
@@ -132,7 +136,31 @@ Required: `ffmpeg`
 
 Some features work better with: `mpv`, `fd-find`, `fish`
 
-## Getting started
+Optional Python extras are grouped under `library[deluxe]`. This includes
+`gallery-dl` (used by `gallery-add`), `internetarchive` (used for Internet
+Archive downloads), and `cdx_toolkit` (used by the `--webcache`/`--wayback`
+CDX extractor):
+
+    pip install "library[deluxe]"
+
+Torrent metadata commands (`torrents-add` and `torrents-dump`) require the
+`libtorrent` Python bindings. They are included by `library[deluxe]` when using
+Python 3.12 or 3.13; Python 3.14 wheels are not available yet, so use Python
+3.13 for torrent support.
+
+If the PyPI wheel is unavailable for your Python or platform, these system
+packages provide the bindings:
+
+* Debian/Ubuntu: `sudo apt install python3-libtorrent`
+* Fedora: `sudo dnf install rb_libtorrent-python3`
+* macOS/Homebrew: `brew install libtorrent-rasterbar`
+
+The Debian, Fedora, and Homebrew packages are built for their system or
+Homebrew Python and may not be importable from a separate uv virtualenv. There
+is no official Scoop package for the Python bindings; on Windows, use Python
+3.13 and `pip install "library[deluxe]"`.
+
+</details>
 
 <details><summary>Local media</summary>
 
@@ -207,7 +235,7 @@ To stop playing press Ctrl+C in either the terminal or mpv
 <details><summary>List all subcommands</summary>
 
     $ library
-    library (v3.2.007; 102 subcommands)
+    library (v3.2.011; 102 subcommands)
 
     Create database subcommands:
     ╭─────────────────┬──────────────────────────────────────────╮
@@ -905,6 +933,14 @@ BTW, for some cols like time_deleted you'll need to specify a where clause so th
         --page-key start --page-start 0 --page-step 50 --fixed-pages 14 --stop-pages-no-match 1 \
         plab.db https://plab/forum/tracker.php?o=(string replace ' ' \n -- 1 4 7 10 15)&s=2&tm=-1&f=(string replace ' ' \n -- 1670 1768 60 1671 1644 1672 1111 508 555 1112 1718 1143 1717 1851 1713 1712 1775 1674 902 1675 36 1830 1803 1831 1741 1676 1677 1780 1110 1124 1784 1769 1793 1797 1804 1819 1825 1836 1842 1846 1857 1861 1867 1451 1788 1789 1792 1798 1805 1820 1826 1837 1843 1847 1856 1862 1868 284 1853 1823 1800 1801 1719 997 1818 1849 1711 1791 1762)
 
+    Recursively add every link on a site (glob excludes prune subtrees from the crawl)
+
+        library links-add links.db --recursive https://example.com/docs/ --path-exclude '*/tags/*'
+
+    Add every archived link for a site from the Internet Archive CDX index
+
+        library links-add links.db --webcache https://example.com/docs/
+
 
 </details>
 
@@ -1238,6 +1274,31 @@ BTW, for some cols like time_deleted you'll need to specify a where clause so th
         library links https://en.wikipedia.org/wiki/List_of_bacon_dishes --path-include https://en.wikipedia.org/wiki/ --after-include famous
         https://en.wikipedia.org/wiki/Omelette
 
+    Download the filtered links. Internet Archive details pages use the Internet Archive downloader and preserve all original files:
+
+        library extract-links --download https://readrare.com/
+
+    Mirror a site recursively. Only links within the same domain and parent directory as the given
+    path are followed; links pointing elsewhere are downloaded but not followed. Includes/excludes
+    accept globs, so you can prune whole subtrees while crawling:
+
+        library links --recursive --download https://example.com/docs/
+        library links -R --download https://example.com/docs/ --path-exclude '*/tag/*' '*/tags/*'
+
+    Extract links from a web archive CDX index instead of crawling live pages. Internet Archive
+    Wayback snapshots are emitted by default. Wayback URLs are collapsed to their original
+    archived URL while crawling: sibling pages that only exist under a different timestamp are
+    still followed, but multiple captures of the same page are collapsed so only one copy of
+    each page is downloaded. Use --webcache-from/--webcache-to to target a time period:
+
+        library links --webcache https://example.com/docs/
+        library links --webcache --recursive --download https://example.com/docs/
+        library links --webcache --webcache-source cc https://example.com/docs/
+        library links --webcache --webcache-from 2020 --webcache-to 2021 https://example.com/docs/
+
+    Common tracking and analytics URLs, including Google Tag Manager and Google Analytics,
+    are excluded by default. Use --path-exclude to replace the default exclusions when needed.
+
     Read from local clipboard and filter out links based on nearby plain text
 
         library links --local-html (cb -t text/html | psub) --after-exclude paranormal spooky horror podcast tech fantasy supernatural lecture sport
@@ -2428,7 +2489,7 @@ Inspired somewhat by https://nikkhokkho.sourceforge.io/?page=FileOptimizer
         | /home/xk/github/xk/lb/.ruff_cache/    |   19.5 kB |     100 |
         | /home/xk/github/xk/lb/.gitattributes  | 119 Bytes |         |
         | /home/xk/github/xk/lb/.mypy_cache/    | 280 Bytes |       4 |
-        | /home/xk/github/xk/lb/.pdm-python     |  15 Bytes |         |
+        | /home/xk/github/xk/lb/.venv            |  15 Bytes |         |
 
     Only include files with a specific depth
 
@@ -2601,6 +2662,17 @@ Inspired somewhat by https://nikkhokkho.sourceforge.io/?page=FileOptimizer
         library download dl.db --include https://www.youtube.com/watch?v=YE7VzlLtp-4
         library download dl.db -s https://www.youtube.com/watch?v=YE7VzlLtp-4  # equivalent
 
+    Recursively download a site (filesystem profile). Only links within the same domain and
+    parent directory as the seed are followed; other links are downloaded but not followed.
+
+        library download links.db --filesystem --recursive --prefix ~/mirror/ https://example.com/docs/
+
+    Download a site's archived pages from the Internet Archive Wayback Machine using the CDX API.
+    `--webcache` implies `--links`; add `--recursive` to also parse the archived pages for
+    further links:
+
+        library download links.db --filesystem --webcache --prefix ~/mirror/ https://example.com/docs/
+
     Maximizing the variety of subdomains
 
         library download photos.db --photos --image --sort "ROW_NUMBER() OVER ( PARTITION BY SUBSTR(m.path, INSTR(m.path, '//') + 2, INSTR( SUBSTR(m.path, INSTR(m.path, '//') + 2), '/') - 1) )"

{library-3.2.7 → library-3.2.11}/library/__main__.py

@@ -2,11 +2,10 @@ import argparse, importlib, os, sys, textwrap
 
 from tabulate import tabulate
 
+from library._version import __version__
 from library.utils import argparse_utils, iterables
 from library.utils.log_utils import log
 
-__version__ = "3.2.007"
-
 progs = {
     "Create database subcommands": {
         "fs_add": "Add local media",

library-3.2.11/library/_version.py

@@ -0,0 +1,25 @@
+# Copyright (c) 2026, Jacob Chapman
+
+from importlib.metadata import PackageNotFoundError
+from importlib.metadata import version as installed_version
+from pathlib import Path
+import tomllib
+
+
+def get_version() -> str:
+    pyproject_path = Path(__file__).resolve().parents[1] / "pyproject.toml"
+    if pyproject_path.is_file():
+        with pyproject_path.open("rb") as pyproject_file:
+            project_version = tomllib.load(pyproject_file)["project"].get("version")
+        if isinstance(project_version, str):
+            return project_version
+        message = f"Missing string project version in {pyproject_path}"
+        raise RuntimeError(message)
+
+    try:
+        return installed_version("library")
+    except PackageNotFoundError as error:
+        raise RuntimeError("Unable to determine the library version") from error
+
+
+__version__ = get_version()

{library-3.2.7 → library-3.2.11}/library/createdb/gallery_backend.py

@@ -2,9 +2,6 @@ import os
 from pathlib import Path
 from types import ModuleType
 
-import gallery_dl
-from gallery_dl.job import Job
-
 from library.mediadb import db_media, db_playlists
 from library.utils import consts, printing, strings
 from library.utils.log_utils import log
@@ -142,52 +139,57 @@ def download(args, m):
     )
 
 
-class UrlJob(Job):
-    resolve = 1  # depth of queue resolution
+def url_job_class(gallery_dl):
+    # gallery_dl is an optional (deluxe) dependency, so the Job base class
+    # can only be resolved once gallery_dl has been imported
+    class UrlJob(gallery_dl.job.Job):
+        resolve = 1  # depth of queue resolution
 
-    def __init__(self, url, parent=None, resolve=None):
-        super().__init__(url, parent)
-        self.results = []
-        self.visited = set()
-        if resolve is not None:
-            self.resolve = resolve
+        def __init__(self, url, parent=None, resolve=None):
+            super().__init__(url, parent)
+            self.results = []
+            self.visited = set()
+            if resolve is not None:
+                self.resolve = resolve
 
-        if self.resolve > 0:
-            self.handle_queue = self.handle_queue_resolve
+            if self.resolve > 0:
+                self.handle_queue = self.handle_queue_resolve
 
-    def handle_url(self, url, kwdict):
-        self.results.append((url, kwdict))
+        def handle_url(self, url, kwdict):
+            self.results.append((url, kwdict))
 
-    def handle_queue(self, url, kwdict):
-        # unresolved queue entry
-        self.results.append((url, kwdict))
+        def handle_queue(self, url, kwdict):
+            # unresolved queue entry
+            self.results.append((url, kwdict))
 
-    def handle_queue_resolve(self, url, kwdict):
-        if url in self.visited:
-            return
-        self.visited.add(url)
+        def handle_queue_resolve(self, url, kwdict):
+            if url in self.visited:
+                return
+            self.visited.add(url)
 
-        cls = kwdict.get("_extractor")
-        if cls:
-            extr = cls.from_url(url)
-        else:
-            extr = self.extractor.find(url)
+            cls = kwdict.get("_extractor")
+            if cls:
+                extr = cls.from_url(url)
+            else:
+                extr = self.extractor.find(url)
 
-        if not extr:
-            self.results.append((url, kwdict))
-            return
+            if not extr:
+                self.results.append((url, kwdict))
+                return
 
-        job = self.__class__(extr, self, self.resolve - 1)
-        job.results = self.results  # shared accumulator
-        job.visited = self.visited  # shared visited set
-        job.run()
+            job = self.__class__(extr, self, self.resolve - 1)
+            job.results = self.results  # shared accumulator
+            job.visited = self.visited  # shared visited set
+            job.run()
+
+    return UrlJob
 
 
 def get_playlist_metadata(args, playlist_path):
     gallery_dl = load_module_level_gallery_dl(args)
 
     added_media_count = 0
-    job = UrlJob(playlist_path)
+    job = url_job_class(gallery_dl)(playlist_path)
     job.run()
 
     is_playlist = len(job.results) > 1

{library-3.2.7 → library-3.2.11}/library/createdb/links_add.py

@@ -130,6 +130,7 @@ To compensate for this the script will only continue fetching pages until there
     )
 
     arggroups.filter_links(parser)
+    arggroups.spider(parser)
 
     arggroups.requests(parser)
     arggroups.selenium(parser)
@@ -254,6 +255,8 @@ def extractor(args, playlist_path):
     new_media = set()
     end_of_playlist = False
     page_limit = args.backfill_pages or args.fixed_pages or args.max_pages
+    if args.recursive or args.webcache:
+        page_limit = 1
 
     if args.confirm_ready:
         if args.selenium:
@@ -283,8 +286,14 @@ def extractor(args, playlist_path):
         page_known = set()
         page_new = {}
         try:
-            for link_dict in extract_links.get_inner_urls(args, page_path):
+            link_iter = (
+                extract_links.crawl(args, [page_path])
+                if args.recursive
+                else extract_links.iter_links(args, page_path)
+            )
+            for link_dict in link_iter:
                 link = link_dict.pop("link")
+                link_dict.pop("mime", None)
 
                 if link == args.stop_link:
                     end_of_playlist = True
@@ -302,7 +311,7 @@ def extractor(args, playlist_path):
                 printing.print_overwrite(f"Page {page_count} link scan: {len(page_new)} new [{len(page_known)} known]")
             print(file=sys.stderr)
 
-            if not (args.backfill_pages or args.fixed_pages):
+            if not args.recursive and not (args.backfill_pages or args.fixed_pages):
                 if (args.stop_known and len(page_known) > args.stop_known) or (
                     args.stop_new and args.stop_new >= len(page_new)
                 ):

{library-3.2.7 → library-3.2.11}/library/mediadb/download.py

@@ -35,6 +35,7 @@ def parse_args():
     arggroups.requests(parser)
     arggroups.selenium(parser)
     arggroups.filter_links(parser)
+    arggroups.spider(parser)
 
     parser.add_argument("--same-domain", action="store_true", help="Choose a random domain to focus on")
 
@@ -88,6 +89,9 @@ def download(args=None) -> None:
     db_playlists.create(args)
     db_media.create(args)
 
+    if args.recursive or args.webcache:
+        args.links = True
+
     m_columns = db_utils.columns(args, "media")
 
     if "limit" in args.defaults and "media" in args.db.table_names() and "webpath" in m_columns:
@@ -118,7 +122,7 @@ def download(args=None) -> None:
         media_printer.media_printer(args, media)
         return
 
-    get_inner_urls = iterables.return_unique(extract_links.get_inner_urls, lambda d: d["link"])
+    get_inner_urls = iterables.return_unique(extract_links.iter_links, lambda d: d["link"])
     if args.safe and args.profile == DBType.image:
         gallery_backend.load_module_level_gallery_dl(args)
     for m in media:
@@ -180,7 +184,11 @@ def download(args=None) -> None:
                 if args.links:
                     dl_paths = []
                     try:
-                        for link_dict in get_inner_urls(args, original_path):
+                        if args.recursive:
+                            link_iter = extract_links.crawl(args, [original_path])
+                        else:
+                            link_iter = get_inner_urls(args, original_path)
+                        for link_dict in link_iter:
                             dl_paths.append(link_dict["link"])
                     except requests.HTTPError as excinfo:
                         log.warning(
@@ -207,7 +215,7 @@ def download(args=None) -> None:
                 for i, dl_path in enumerate(dl_paths):
                     error = None
                     try:
-                        local_path = web.download_url(args, dl_path)
+                        local_path = web.download_link(args, dl_path)
                     except RuntimeError as excinfo:
                         local_path = None
                         error = str(excinfo)

{library-3.2.7 → library-3.2.11}/library/mediafiles/images_to_pdf.py

@@ -112,7 +112,7 @@ def gen_arg_groups(args):
             else:
                 log.warning("No images found in %s", path)
         elif path_utils.ext(path) in consts.ARCHIVE_EXTENSIONS:
-            archive_dir = processes.unar_delete(path)
+            archive_dir, _ = processes.unar_delete(path)
             archive_image_paths = shell_utils.rglob(str(archive_dir), DEFAULT_EXTENSIONS, quiet=True)[0]
             if archive_image_paths:
                 yield archive_image_paths

{library-3.2.7 → library-3.2.11}/library/mediafiles/process_image.py

@@ -52,7 +52,7 @@ def process_path(args, path) -> str | None:
         if args.simulate:
             log.info("Extracting images %s", path)
         else:
-            archive_dir = processes.unar_delete(path)
+            archive_dir, _ = processes.unar_delete(path)
             image_paths = shell_utils.rglob(str(archive_dir), consts.IMAGE_EXTENSIONS, quiet=True)[0]
             for p in image_paths:
                 process_path(args, p)