mediaferry 0.2.1__tar.gz

mediaferry-0.2.1/LICENSE

@@ -0,0 +1,112 @@
+Creative Commons Legal Code
+
+CC0 1.0 Universal
+
+Statement of Purpose
+
+The laws of most jurisdictions throughout the world automatically confer
+exclusive Copyright and Related Rights (defined below) upon the creator
+and subsequent owner(s) (each and all, an "owner") of an original work of
+authorship and/or a database (each, a "Work").
+
+Certain owners wish to permanently relinquish those rights to a Work for
+the purpose of contributing to a commons of creative, cultural and
+scientific works ("Commons") that the public can reliably and without fear
+of later claims of infringement build upon, modify, incorporate in other
+works, reuse and redistribute as freely as possible in any form whatsoever
+and for any purposes, including without limitation commercial purposes.
+These owners may contribute to the Commons to promote the ideal of a free
+culture and the further production of creative, cultural and scientific
+works, or to gain reputation or greater distribution for their Work in
+part through the use and efforts of others.
+
+For these and/or other purposes and motivations, and without any
+expectation of additional consideration or compensation, the person
+associating CC0 with a Work (the "Affirmer"), to the extent that he or she
+is an owner of Copyright and Related Rights in the Work, voluntarily
+elects to apply CC0 to the Work and publicly distribute the Work under its
+terms, with knowledge of his or her Copyright and Related Rights in the
+Work and the meaning and intended legal effect of CC0 on those rights.
+
+1. Copyright and Related Rights. A Work made available under CC0 may be
+protected by copyright and related or neighboring rights ("Copyright and
+Related Rights"). Copyright and Related Rights include, but are not
+limited to, the following:
+
+  i. the right to reproduce, adapt, distribute, perform, display,
+     communicate, and translate a Work;
+ ii. moral rights retained by the original author(s) and/or performer(s);
+iii. publicity and privacy rights pertaining to a person's image or
+     likeness depicted in a Work;
+ iv. rights protecting against unfair competition in regards to a Work,
+     subject to the limitations in paragraph 4(a), below;
+  v. rights protecting the extraction, dissemination, use and reuse of data
+     in a Work;
+ vi. database rights (such as those arising under Directive 96/9/EC of the
+     European Parliament and of the Council of 11 March 1996 on the legal
+     protection of databases, and under any national implementation
+     thereof, including any amended or successor version of such
+     directive); and
+vii. other similar, equivalent or corresponding rights throughout the
+     world based on applicable law or treaty, and any national
+     implementations thereof.
+
+2. Waiver. To the greatest extent permitted by, but not in contravention
+of, applicable law, Affirmer hereby overtly, fully, permanently,
+irrevocably and unconditionally waives, abandons, and surrenders all of
+Affirmer's Copyright and Related Rights and associated claims and causes
+of action, whether now known or unknown (including existing as well as
+future claims and causes of action), in the Work (i) in all territories
+worldwide, (ii) for the maximum duration provided by applicable law or
+treaty (including future time extensions), (iii) in any current or future
+medium and for any number of copies, and (iv) for any purpose whatsoever,
+including without limitation commercial, advertising or promotional
+purposes (the "Waiver"). Affirmer makes the Waiver for the benefit of each
+member of the public at large and to the detriment of Affirmer's heirs and
+successors, fully intending that such Waiver shall not be subject to
+revocation, rescission, cancellation, termination, or any other legal or
+equitable action to disrupt the quiet enjoyment of the Work by the public
+as contemplated by Affirmer's express Statement of Purpose.
+
+3. Public License Fallback. Should any part of the Waiver for any reason
+be judged legally invalid or ineffective under applicable law, then the
+Waiver shall be preserved to the maximum extent permitted taking into
+account Affirmer's express Statement of Purpose. In addition, to the
+extent the Waiver is so judged Affirmer hereby grants to each affected
+person a royalty-free, non transferable, non sublicensable, non exclusive,
+irrevocable and unconditional license to exercise Affirmer's Copyright and
+Related Rights in the Work (i) in all territories worldwide, (ii) for the
+maximum duration provided by applicable law or treaty (including future
+time extensions), (iii) in any current or future medium and for any number
+of copies, and (iv) for any purpose whatsoever, including without
+limitation commercial, advertising or promotional purposes (the
+"License"). The License shall be deemed effective as of the date CC0 was
+applied by Affirmer to the Work. Should any part of the License for any
+reason be judged legally invalid or ineffective under applicable law, such
+partial invalidity or ineffectiveness shall not invalidate the remainder
+of the License, and in such case Affirmer hereby affirms that he or she
+will not (i) exercise any of his or her remaining Copyright and Related
+Rights in the Work or (ii) assert any associated claims and causes of
+action with respect to the Work, in either case contrary to Affirmer's
+express Statement of Purpose.
+
+4. Limitations and Disclaimers.
+
+ a. No trademark or patent rights held by Affirmer are waived, abandoned,
+    surrendered, licensed or otherwise affected by this document.
+ b. Affirmer offers the Work as-is and makes no representations or
+    warranties of any kind concerning the Work, express, implied,
+    statutory or otherwise, including without limitation warranties of
+    title, merchantability, fitness for a particular purpose, non
+    infringement, or the absence of latent or other defects, accuracy, or
+    the present or absence of errors, whether or not discoverable, all to
+    the greatest extent permissible under applicable law.
+ c. Affirmer disclaims responsibility for clearing rights of other persons
+    that may apply to the Work or any use thereof, including without
+    limitation any person's Copyright and Related Rights in the Work.
+    Further, Affirmer disclaims responsibility for obtaining any necessary
+    consents, permissions or other rights required for any use of the
+    Work.
+ d. Affirmer understands and acknowledges that Creative Commons is not a
+    party to this document and has no duty or obligation with respect to
+    this CC0 or use of the Work.

mediaferry-0.2.1/PKG-INFO

@@ -0,0 +1,40 @@
+Metadata-Version: 2.4
+Name: mediaferry
+Version: 0.2.1
+Summary: yt-dlp download management tool
+License-File: LICENSE
+Author: DigitalDragon
+Requires-Python: >=3.8
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Requires-Dist: dataclasses-json
+Requires-Dist: humanize
+Requires-Dist: internetarchive (>=5.0.3)
+Requires-Dist: yt-dlp
+Project-URL: Repository, https://github.com/DigitalDwagon/MediaFerry
+Description-Content-Type: text/markdown
+
+mediaferry is a simple CLI interface to download videos with yt-dlp in a consistent format that can be later uploaded to the Internet Archive with a tool like `dartboard`
+
+
+```
+usage: mediaferry [-h] [--verbose] [--force] [--metadata METADATA] [--cookies COOKIES] url
+
+MediaFerry
+
+positional arguments:
+  url                  URL of the media to download
+
+options:
+  -h, --help           show this help message and exit
+  --verbose            Enable verbose output
+  --force              Force download even if an archive.org item already exists
+  --metadata METADATA  Add metadata as key:value pairs
+  --cookies COOKIES    Path to a cookies.txt file to use for downloading
+```

mediaferry-0.2.1/README.md

@@ -0,0 +1,18 @@
+mediaferry is a simple CLI interface to download videos with yt-dlp in a consistent format that can be later uploaded to the Internet Archive with a tool like `dartboard`
+
+
+```
+usage: mediaferry [-h] [--verbose] [--force] [--metadata METADATA] [--cookies COOKIES] url
+
+MediaFerry
+
+positional arguments:
+  url                  URL of the media to download
+
+options:
+  -h, --help           show this help message and exit
+  --verbose            Enable verbose output
+  --force              Force download even if an archive.org item already exists
+  --metadata METADATA  Add metadata as key:value pairs
+  --cookies COOKIES    Path to a cookies.txt file to use for downloading
+```

mediaferry-0.2.1/mediaferry/__version__.py

@@ -0,0 +1,4 @@
+import importlib.metadata
+
+version : str = importlib.metadata.version(__package__ or __name__)
+yt_dlp_version : str = importlib.metadata.version("yt-dlp")

mediaferry-0.2.1/mediaferry/config.py

@@ -0,0 +1,9 @@
+from dataclasses import dataclass, field
+
+
+@dataclass
+class Config:
+    force: bool = False
+    verbose: bool = False
+    metadata: dict = field(default_factory=dict)
+    cookies: str = None

mediaferry-0.2.1/mediaferry/download.py

@@ -0,0 +1,180 @@
+import json
+import os
+import re
+from datetime import datetime
+from urllib.parse import urlparse
+
+import internetarchive
+import humanize
+import yt_dlp
+
+from mediaferry.__version__ import version, yt_dlp_version
+
+class Media:
+    def __init__(self, media_info: dict):
+        self.url = media_info["webpage_url"]
+        self.id = media_info.get("display_id", media_info.get("id"))
+        self.extractor = media_info["extractor"]
+        self.media_info = media_info
+
+
+    def get_identifier(self):
+        return re.sub(r'[^\w-]', "-", f"{self.extractor}-{self.id}")
+
+    def internetarchive_item_exists(self):
+        item = internetarchive.get_item(self.get_identifier())
+        return item.exists
+
+    def download(self, config):
+        print(f"Downloading {self.url} ({self.get_identifier()})...")
+        if not config.force and self.internetarchive_item_exists():
+            print(f"Item {self.get_identifier()} already exists on archive.org. If you want to download it anyways, use --force.")
+            print(f"https://archive.org/details/{self.get_identifier()}")
+            return False
+
+        directory = os.path.abspath(self.get_identifier())
+        os.makedirs(directory, exist_ok=True)
+
+        options = get_ytdlp_options(config, self.get_identifier())
+        options["progress_hooks"] = [progress_hook]
+        with yt_dlp.YoutubeDL(options) as ydl:
+            status_code = ydl.download([self.url])
+
+            if status_code != 0:
+                print(f"Error downloading {self.url} (exit status {status_code}).")
+
+        print(f"Downloaded {self.url} to {self.get_identifier()}.")
+        print("Generating metadata...")
+        metadata = self.get_internetarchive_metadata()
+        print(json.dumps(metadata, indent=4))
+
+        # Safety check - if there isn't an MKV, the video wasn't downloaded.
+        if not any(fname.endswith(".mkv") for fname in os.listdir(directory)):
+            print(f"Error: No MKV file found in {directory}. Did the download or remux fail?")
+            with open(os.path.join(directory, "__ia_meta.json.failed_download"), "a") as f:
+                f.write(json.dumps(metadata, indent=4))
+            return False
+
+        with open(os.path.join(directory, "__ia_meta.json"), "a") as f:
+            f.write(json.dumps(metadata, indent=4))
+
+        # delete empty description files, consistent with tubeup
+        for filename in os.listdir(directory):
+            if filename.endswith(".description"):
+                filepath = os.path.join(directory, filename)
+                if os.path.getsize(filepath) == 0:
+                    os.remove(filepath)
+                    print(f"Removed empty description file {filepath}")
+
+
+
+
+
+    def get_internetarchive_metadata(self):
+        ia_metadata = {
+            'scanner': [f'mediaferry (v{version})', f'yt-dlp (v{yt_dlp_version})'],
+            'scandate': datetime.now().isoformat(),
+            'collection': 'opensource_movies',
+            'mediatype': 'movies'
+        }
+
+        # yt-dlp metadata -> IA item metadata
+        alias_keys = {
+            'title': 'title',
+            'description': 'description',
+            'webpage_url': 'originalurl',
+            'uploader': 'creator',
+            'uploader_url': 'channel',
+            'channel_url': 'channel',
+        }
+
+        for key, value in alias_keys.items():
+            if key in self.media_info and not value in ia_metadata:
+                ia_metadata[value] = re.sub('\r?\n', '<br>', self.media_info[key])
+
+        if urlparse(self.url).netloc == 'soundcloud.com':
+            ia_metadata['collection'] = 'opensource_audio'
+            ia_metadata['mediatype'] = 'audio'
+
+        if self.media_info.get('extractor_key') == 'TwitchClips' and self.media_info.get('creator', False):
+            ia_metadata['creator'] = self.media_info.get('creator')
+
+        if 'upload_date' in self.media_info:
+            date = datetime.strptime(self.media_info['upload_date'], '%Y%m%d')
+            ia_metadata["date"] = date.strftime('%Y-%m-%d')
+
+        subject = ["video", self.media_info.get('extractor_key')]
+        if 'categories' in self.media_info:
+            subject.extend(self.media_info['categories'])
+        if 'tags' in self.media_info:
+            subject.extend(self.media_info['tags'])
+        while len(";".join(subject).encode("utf-8")) > 255:
+            subject.pop()
+        ia_metadata['subject'] = ";".join(subject)
+
+        return ia_metadata
+
+
+def progress_hook(d):
+    if True:
+        return
+    # causing errors
+
+    if d['status'] == 'downloading':
+        print("Downloading %s: %s/%s at %s/s, %s" % (
+            d.get('filename', '(unknown)'),
+            humanize.naturalsize(d.get('downloaded_bytes', 0)),
+            humanize.naturalsize(d.get('total_bytes', d.get('total_bytes_estimate', 0))),
+            humanize.naturalsize(d.get('speed', 0)),
+            d.get('eta', 'unknown')
+        ))
+
+    if d['status'] == 'finished':
+        print("Done downloading %s: %s at %s/s" % (
+            d.get('filename', '(unknown)'),
+            humanize.naturalsize(d.get('total_bytes', 0)),
+            humanize.naturalsize(d.get('total_bytes', 0) / d.get('elapsed', 1))
+        ))
+
+    if d['status'] == 'error':
+        print(json.dumps(d))
+        print("Error downloading...")
+
+
+def get_ytdlp_options(config, directory) -> dict:
+    ytdlp_options =  {
+        "outtmpl": os.path.join(directory, "%(id)s.%(ext)s"),
+        "restrictfilenames": True,
+        "quiet": not config.verbose,
+        "verbose": config.verbose,
+        "progress_with_newline": True,
+        "forcetitle": True,
+        "continuedl": True,
+        "retries": 3,
+        "fragment_retries": 3,
+        "forcejson": False,
+        "writeinfojson": True,
+        "writedescription": True,
+        "writethumbnail": True,
+        "writeannotations": True,
+        "writesubtitles": True,
+        "allsubtitles": True,
+        "ignoreerrors": True, # Allow full-channel downloads to continue even if some videos fail
+        "fixup": "detect_or_warn",
+        "nooverwrites": True, # Don't re-download files that already exist (ex. manual retry)
+        "consoletitle": True,
+        "prefer_ffmpeg": True,
+        "call_home": False,
+        "cookiefile": config.cookies,
+        "merge_output_format": "mkv"
+        #"proxy": args.proxy,
+        #"username": args.username,
+        #"password": args.password,
+    }
+
+    for key, value in list(ytdlp_options.items()):
+        if value is None:
+            del ytdlp_options[key]
+
+    return ytdlp_options
+

mediaferry-0.2.1/mediaferry/main.py

@@ -0,0 +1,53 @@
+import argparse
+import json
+import os
+from collections import defaultdict
+
+from yt_dlp import YoutubeDL
+
+from mediaferry.config import Config
+from mediaferry.download import Media, get_ytdlp_options
+
+
+def main():
+
+    parser = argparse.ArgumentParser(description="MediaFerry")
+    parser.add_argument("--verbose", action="store_true", help="Enable verbose output")
+    parser.add_argument("--force", action="store_true", help="Force download even if an archive.org item already exists")
+    parser.add_argument("url", help="URL of the media to download")
+    parser.add_argument("--metadata", action=ParseMetadata, default=defaultdict(list), help="Add metadata as key:value pairs")
+    # yt-dlp options
+    parser.add_argument("--cookies", type=str, default=None, help="Path to a cookies.txt file to use for downloading")
+    args = parser.parse_args()
+
+    config_args = {key: value for key, value in vars(args).items() if key != "url"}
+    config = Config(**config_args)
+    download_media(args.url, config)
+
+def download_media(url, config):
+    print(f"Discovering {url}")
+    with YoutubeDL(get_ytdlp_options(config, ".")) as ydl:
+        media_info = ydl.sanitize_info(ydl.extract_info(url, download=False))
+
+        if media_info.get("_type", "video") == "playlist":
+            print("Found playlist!")
+            for entry in media_info["entries"]:
+                print(f"Downloading playlist entry {entry['webpage_url']}")
+                media: Media = Media(entry)
+                media.download(config)
+        else:
+            media: Media = Media(media_info)
+            media.download(config)
+
+class ParseMetadata(argparse.Action):
+    def __call__(self, parser, namespace, values, option_string=None):
+        metadata = getattr(namespace, self.dest, defaultdict(list))
+        key, value = values.split(":", 1)
+        metadata[key].append(value)
+        setattr(namespace, self.dest, metadata)
+
+
+
+
+if __name__ == "__main__":
+    main()

mediaferry-0.2.1/pyproject.toml

@@ -0,0 +1,25 @@
+[build-system]
+requires = ["poetry-core>=2.0.0,<3.0.0"]
+build-backend = "poetry.core.masonry.api"
+
+[project]
+name = "mediaferry"
+version = "0.2.1"
+authors = [
+    {name = "DigitalDragon"}
+]
+description = "yt-dlp download management tool"
+readme = "README.md"
+requires-python = ">=3.8"
+dependencies = [
+    "internetarchive>=5.0.3",
+    "dataclasses-json",
+    "yt-dlp",
+    "humanize"
+]
+
+[project.scripts]
+mediaferry = "mediaferry.main:main"
+
+[project.urls]
+repository = "https://github.com/DigitalDwagon/MediaFerry"