rcdl 3.0.0b20__tar.gz

rcdl-3.0.0b20/PKG-INFO

@@ -0,0 +1,122 @@
+Metadata-Version: 2.4
+Name: rcdl
+Version: 3.0.0b20
+Summary: Coomer/Kemono CLI Downloader
+Keywords: downloader,video,media
+Author: Anonymous
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Dist: click>=8.2
+Requires-Dist: requests>=2.32
+Requires-Dist: pathvalidate==3.3.1
+Requires-Dist: rich==14.2.0
+Requires-Dist: streamlit==1.52.2
+
+# RCDL
+
+Riton Coomer Download Manager  
+`rcdl` is a tool to automatically download the videos of your favorites creators from [coomer.st](https://coomer.st) and [kemono.cr](https://kemono.cr)
+
+
+## Install
+### Dependencies
+- [yt-dlp](https://github.com/yt-dlp/yt-dlp)
+- [aria2](https://github.com/aria2/aria2)
+- [ffmpeg](https://www.ffmpeg.org/download.html) (Only for `fuse` command)  
+- [HandBrakeCLI](https://handbrake.fr/docs/en/latest/cli/cli-options.html) (Only for `opti` command)
+Recommended install:
+```bash
+pipx install yt-dlp streamlit
+sudo apt update
+sudo apt install aria2 ffmpeg handbrake-cli python3-tk
+```
+### Install RCDL
+It is recommended to use `pipx` to install `rcdl`
+```bash
+pipx install rcdl
+```
+or else:  
+```bash
+pip install rcdl
+```
+
+## How to use
+
+Run the CLI with:
+
+```bash
+rcdl --help
+```
+
+By default all files will live in `~/Videos/rcdl/`. Cache, configuration and log file will be in a hidden `rcdl/.cache/` folder.
+
+Main function:  
+```bash
+rcdl refresh    # look creators.json and find all possible videos
+rcdl dlsf       # download all found videos
+rcdl discover   # Discover new creator (WIP)
+rcdl opti       # Optimized video to reduce disk storage usage
+rcdl fuse       # Fuse all videos within a same post if they are fully downloaded
+```
+
+Manage creators:  
+```bash
+rcdl list   # list all current creators
+rcdl add [URL]
+rcdl add [service]/[creator_id]
+rcdl remove [creator_id]
+```
+
+Helper function:  
+```bash
+rcdl status     # give number of entry in the database per tables and status
+rcdl clean --all      # remove all partially downloaded file, external dependencies cache, etc...
+rcdl show-config        # print all config var and theirs value (paths, etc...)
+```
+
+### Settings
+Default settings file:
+```toml
+[app]
+default_max_page = 10
+max_fail_count = 7
+timeout = 10
+
+[fuse]
+max_width = 1920
+max_height = 1080
+fps = 30
+preset = "veryfast"
+threads = 0
+
+[paths]
+base_dir = "~/Videos/rcdl"
+handbrake_run_cmd = "HandBrakeCLI"
+```
+
+In `rcdl/.cache/config.toml`:
+```toml
+[paths]
+handbrake_run_cmd = "HandBrakeCLI"  # if installed via apt
+handbrake_run_cmd = "flatpak run --command=HandBrakeCLI fr.handbrake.ghb"   # if installed via flatpak
+```
+
+## Dev
+### Install
+```bash
+git clone https://github.com/ritonun/cdl.git rcdl
+cd rcdl
+python3 -m venv .venv
+source .venv/bin/activate
+pip install -e .
+```
+
+## Deploy
+```bash
+python3 -m pip install --upgrade build
+python3 -m pip install --upgrade twine
+```
+

rcdl-3.0.0b20/README.md

@@ -0,0 +1,104 @@
+# RCDL
+
+Riton Coomer Download Manager  
+`rcdl` is a tool to automatically download the videos of your favorites creators from [coomer.st](https://coomer.st) and [kemono.cr](https://kemono.cr)
+
+
+## Install
+### Dependencies
+- [yt-dlp](https://github.com/yt-dlp/yt-dlp)
+- [aria2](https://github.com/aria2/aria2)
+- [ffmpeg](https://www.ffmpeg.org/download.html) (Only for `fuse` command)  
+- [HandBrakeCLI](https://handbrake.fr/docs/en/latest/cli/cli-options.html) (Only for `opti` command)
+Recommended install:
+```bash
+pipx install yt-dlp streamlit
+sudo apt update
+sudo apt install aria2 ffmpeg handbrake-cli python3-tk
+```
+### Install RCDL
+It is recommended to use `pipx` to install `rcdl`
+```bash
+pipx install rcdl
+```
+or else:  
+```bash
+pip install rcdl
+```
+
+## How to use
+
+Run the CLI with:
+
+```bash
+rcdl --help
+```
+
+By default all files will live in `~/Videos/rcdl/`. Cache, configuration and log file will be in a hidden `rcdl/.cache/` folder.
+
+Main function:  
+```bash
+rcdl refresh    # look creators.json and find all possible videos
+rcdl dlsf       # download all found videos
+rcdl discover   # Discover new creator (WIP)
+rcdl opti       # Optimized video to reduce disk storage usage
+rcdl fuse       # Fuse all videos within a same post if they are fully downloaded
+```
+
+Manage creators:  
+```bash
+rcdl list   # list all current creators
+rcdl add [URL]
+rcdl add [service]/[creator_id]
+rcdl remove [creator_id]
+```
+
+Helper function:  
+```bash
+rcdl status     # give number of entry in the database per tables and status
+rcdl clean --all      # remove all partially downloaded file, external dependencies cache, etc...
+rcdl show-config        # print all config var and theirs value (paths, etc...)
+```
+
+### Settings
+Default settings file:
+```toml
+[app]
+default_max_page = 10
+max_fail_count = 7
+timeout = 10
+
+[fuse]
+max_width = 1920
+max_height = 1080
+fps = 30
+preset = "veryfast"
+threads = 0
+
+[paths]
+base_dir = "~/Videos/rcdl"
+handbrake_run_cmd = "HandBrakeCLI"
+```
+
+In `rcdl/.cache/config.toml`:
+```toml
+[paths]
+handbrake_run_cmd = "HandBrakeCLI"  # if installed via apt
+handbrake_run_cmd = "flatpak run --command=HandBrakeCLI fr.handbrake.ghb"   # if installed via flatpak
+```
+
+## Dev
+### Install
+```bash
+git clone https://github.com/ritonun/cdl.git rcdl
+cd rcdl
+python3 -m venv .venv
+source .venv/bin/activate
+pip install -e .
+```
+
+## Deploy
+```bash
+python3 -m pip install --upgrade build
+python3 -m pip install --upgrade twine
+```

rcdl-3.0.0b20/pyproject.toml

@@ -0,0 +1,29 @@
+[project]
+name = "rcdl"
+version = "3.0.0b20"
+description = "Coomer/Kemono CLI Downloader"
+requires-python = ">=3.12"
+dependencies = [
+    "click>=8.2",
+    "requests>=2.32",
+    "pathvalidate==3.3.1",
+    "rich==14.2.0",
+    "streamlit==1.52.2"
+]
+authors = [
+    {name = "Anonymous"},
+]
+readme = "README.md"
+keywords = ["downloader", "video", "media"]
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "License :: OSI Approved :: MIT License",
+    "Operating System :: OS Independent",
+]
+
+[project.scripts]
+rcdl = "rcdl.__main__:cli"
+
+[build-system]
+requires = ["flit_core<4"]
+build-backend = "flit_core.buildapi"

rcdl-3.0.0b20/rcdl/__init__.py

@@ -0,0 +1,10 @@
+# __init__.py
+
+"""
+Init file for package structure.
+Get the version from pyproject.toml version tag
+"""
+
+from importlib.metadata import version
+
+__version__ = version("rcdl")

rcdl-3.0.0b20/rcdl/__main__.py

@@ -0,0 +1,37 @@
+# __main__.py
+
+"""
+__main__: entry in the programm
+Setup logging, create files/folders structures, check dependencies,
+init Config global variables, init database,
+Then call cli group
+"""
+
+import logging
+
+from rcdl.core.config import Config, setup_logging, check_dependencies
+
+# setup file structure
+Config.ensure_dirs()
+Config.ensure_files()
+
+# load config file settings
+Config.load_config()
+
+# setup logging
+setup_logging(Config.LOG_FILE, level=0)
+
+# check dependencies
+check_dependencies()
+
+logging.info("--- INIT ---")
+logging.info("Logger initialized")
+
+# init database
+from rcdl.core.db import DB  # noqa: E402
+
+db = DB()
+db.init_database()
+db.close()
+
+from rcdl.interface.cli import cli  # noqa: E402, F401

rcdl-3.0.0b20/rcdl/core/adapters.py

@@ -0,0 +1,241 @@
+# core/adapters.py
+
+"""Convert [Any] into proper Models from models.py"""
+
+import json
+import sqlite3
+from dataclasses import fields
+
+from rcdl.core import parser
+from rcdl.interface.ui import UI
+from rcdl.core.models import Post, Media, Status, FusedMedia
+from rcdl.utils import get_date_now, get_json_hash
+
+VALID_POST_KEYS = set(
+    [
+        "id",
+        "user",
+        "service",
+        "title",
+        "substring",
+        "published",
+        "file",
+        "attachments",
+    ]
+)
+
+
+def _compute_json_metadata(raw: dict) -> tuple[str, str, str]:
+    """From a json dict, return:
+    - raw_json: str
+    - json_hash: str
+    - fetched_at str (datetime)
+    """
+    raw_json, json_hash = get_json_hash(raw)
+    fetched_at = get_date_now()
+    return raw_json, json_hash, fetched_at
+
+
+def json_posts_to_posts(posts: list[dict]) -> list[Post]:
+    """Convert a list of json post (dict) into a list of Post model
+    Ignore if conversion failed"""
+    formatted_posts = []
+    for post in posts:
+        p = json_post_to_post(post)
+        if p is not None:
+            formatted_posts.append(p)
+    return formatted_posts
+
+
+def json_post_to_post(post: dict) -> Post | None:
+    """Convert a json post (dict) into a Post model
+    or return None if covnersion failed"""
+    post_keys = set(post)
+    if post_keys != VALID_POST_KEYS:
+        UI.error(
+            f"Post id {post.get('id')} of {post.get('user')} "
+            f"has invalid schema. "
+            f"Missing: {VALID_POST_KEYS - post_keys}, "
+            f"Extra: {post_keys - VALID_POST_KEYS}"
+        )
+        return None
+
+    try:
+        domain = parser.get_domain(post["service"])
+        raw_json, json_hash, fetched_at = _compute_json_metadata(post)
+        return Post(
+            **post,
+            domain=domain,
+            json_hash=json_hash,
+            raw_json=raw_json,
+            fetched_at=fetched_at,
+        )
+    except TypeError as e:
+        UI.error(
+            f"Post id {post.get('id')} from {post.get('user')} could not be parsed: {e}"
+        )
+        return None
+
+
+def row_to_post(row: sqlite3.Row) -> Post | None:
+    """Convert a sqlite3 row into a Post model.
+    Return None if conversion failed"""
+    try:
+        raw = json.loads(row["raw_json"])
+        return Post(
+            id=row["id"],
+            user=row["user"],
+            service=row["service"],
+            domain=row["domain"],
+            published=row["published"],
+            json_hash=row["json_hash"],
+            raw_json=row["raw_json"],
+            fetched_at=row["fetched_at"],
+            title=raw["title"],
+            substring=raw["substring"],
+            file=raw["file"],
+            attachments=raw["attachments"],
+        )
+    except KeyError as e:
+        UI.error(
+            f"KeyError: Failed to convert {row['id']} (row_id) into Post model due to: {e}"
+        )
+        return None
+    except TypeError as e:
+        UI.error(
+            f"TypeError: Failed to convert {row['id']} (row_id) into Post model due to: {e}"
+        )
+        return None
+    except ValueError as e:
+        UI.error(
+            f"ValueError/JSONDecodeError: Failed to convert "
+            f"{row['id']} (row_id) into Post model due to: {e}"
+        )
+        return None
+
+
+def rows_to_posts(rows: list[sqlite3.Row]) -> list[Post]:
+    """Convert a list of sqlite3 rows. Return a list of Post model.
+    Ignore the row if conversion fail"""
+    posts: list[Post] = []
+    for row in rows:
+        post = row_to_post(row)
+        if post is not None:
+            posts.append(post)
+
+    if len(posts) != len(rows):
+        UI.error(
+            f"From {len(rows)} rows, only converted {len(posts)}."
+            f" {len(rows) - len(posts)} error."
+        )
+
+    return posts
+
+
+def row_to_media(row: sqlite3.Row) -> Media | None:
+    """Convert a sqlite3 row into a Media model.
+    Return None if conversion failed"""
+    try:
+        # create a dict to hold column of row that are present in Media.
+        # Ignore column (like default autoincrement ID) that are not a field in Media
+        media_data = {}
+        for field in fields(Media):
+            field_name = field.name
+            if field_name in row.keys():
+                value = row[field_name]
+                if field_name == "status" and value is not None:
+                    value = Status(value)
+                media_data[field_name] = value
+        return Media(**media_data)
+    except (KeyError, TypeError, ValueError) as e:
+        UI.error(
+            f"Key/Type/Value Error: Failed to convert row {row['id']} into Post model due to {e}"
+        )
+        return None
+
+
+def rows_to_medias(rows: list[sqlite3.Row]) -> list[Media]:
+    """Convert a list of sqlite3 rows. Return a list of Media model.
+    Ignore row if conversion failed"""
+    medias: list[Media] = []
+    for row in rows:
+        media = row_to_media(row)
+        if media is not None:
+            medias.append(media)
+
+    if len(medias) != len(rows):
+        UI.error(
+            f"From {len(rows)} rows, only converted {len(medias)}."
+            f" {len(rows) - len(medias)} error."
+        )
+
+    return medias
+
+
+def row_to_fused_media(row: sqlite3.Row) -> FusedMedia | None:
+    """Convert a sqlite3 row into a FusedMedia model.
+    Return None if conversion fail"""
+    if row is None:
+        return None
+    try:
+        fuses_data = {}
+        for field in fields(FusedMedia):
+            field_name = field.name
+            if field_name in row.keys():
+                value = row[field_name]
+                if field_name == "status" and value is not None:
+                    value = Status(value)
+                fuses_data[field_name] = value
+        return FusedMedia(**fuses_data)
+    except (KeyError, TypeError, ValueError) as e:
+        UI.error(
+            f"Key/Type/Value Error: Failed to convert row "
+            f"{row['id']} into FusedMedia model due to {e}"
+        )
+        return None
+
+
+def rows_to_fuses(rows: list[sqlite3.Row]) -> list[FusedMedia]:
+    """Convert a lsit of sqlite3 rows into a list of FusedMedia model
+    Ignore row if conversion failed"""
+    fuses: list[FusedMedia] = []
+    for row in rows:
+        fuse = row_to_fused_media(row)
+        if fuse is not None:
+            fuses.append(fuse)
+
+    if len(fuses) != len(rows):
+        UI.error(
+            f"From {len(rows)} rows, only converted {len(fuses)}."
+            f" {len(rows) - len(fuses)} error."
+        )
+
+    return fuses
+
+
+def post_to_videos(post: Post) -> list[Media]:
+    """Extract a list of Media model from a Post model"""
+    json_post = json.loads(post.raw_json)
+
+    urls = parser.extract_video_urls(json_post)
+    sequence = 0
+    medias: list[Media] = []
+    for url in urls:
+        medias.append(
+            Media(
+                post_id=post.id,
+                service=post.service,
+                url=url,
+                duration=0.0,
+                sequence=sequence,
+                status=Status.PENDING,
+                checksum="",
+                file_path=parser.get_filename(json_post, url),
+                created_at="",
+                updated_at="",
+                file_size=0,
+                fail_count=0,
+            )
+        )
+        sequence += 1
+    return medias

rcdl-3.0.0b20/rcdl/core/api.py

@@ -0,0 +1,76 @@
+# core/api.py
+
+"""Build real URL for api request"""
+
+from rcdl.core.models import Creator
+
+
+class URL:
+    """Build real URL for api request"""
+
+    DOMAINS_BASE_URL = {
+        "coomer": "https://coomer.st/api/v1/",
+        "kemono": "https://kemono.cr/api/v1/",
+    }
+
+    @staticmethod
+    def get_base_url(domain: str) -> str:
+        """Return https://domain.com"""
+        if domain not in URL.DOMAINS_BASE_URL:
+            raise KeyError(f"{domain} not in known domains urls")
+        return URL.DOMAINS_BASE_URL[domain]
+
+    @staticmethod
+    def get_post_revision(creator: Creator, post_id) -> str:
+        """Return post revision url"""
+        return (
+            f"{URL.get_base_url(creator.domain)}{creator.service}"
+            f"/user/{creator.id}/post/{post_id}/revisions"
+        )
+
+    @staticmethod
+    def get_headers() -> dict:
+        """Return necessary request header for successful request"""
+        return {
+            "User-Agent": (
+                "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
+                "AppleWebKit/537.36 (KHTML, like Gecko) "
+                "Chrome/117.0 Safari/537.36"
+            ),
+            "Accept": "text/css",
+        }
+
+    @staticmethod
+    def get_url_from_file(domain: str, path_url: str):
+        """Add path_url to based domain url"""
+        if domain == "coomer":
+            return f"https://coomer.st{path_url}"
+        if domain == "kemono":
+            return f"https://kemono.cr{path_url}"
+
+        raise ValueError(
+            f"Domain {domain} is not an accepted value/does not exist. "
+            f"Please check your creators.json file"
+        )
+
+    @staticmethod
+    def add_params(url: str, params: dict):
+        """Create all parameters string (key=params&key=...)"""
+        url += "?"
+        for key in params:
+            url += f"{key}={params[key]}&"
+        return url[:-1]
+
+    @staticmethod
+    def get_creator_post_wo_param(creator: Creator) -> str:
+        """Get creator post without parameters"""
+        return (
+            f"{URL.get_base_url(creator.domain)}{creator.service}"
+            f"/user/{creator.id}/posts"
+        )
+
+    @staticmethod
+    def get_posts_page_url_wo_param():
+        """Get posts page without parameters -> use in tag search"""
+        domain = URL.DOMAINS_BASE_URL["coomer"]
+        return f"{domain}posts"