twotone 1.0.0__tar.gz

twotone-1.0.0/LICENSE

@@ -0,0 +1,8 @@
+
+Copyright 2023 Michał Walenciak
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

twotone-1.0.0/PKG-INFO

@@ -0,0 +1,27 @@
+Metadata-Version: 2.4
+Name: twotone
+Version: 1.0.0
+Summary: Tools for batch video file manipulations
+Requires-Python: >=3.10
+License-File: LICENSE
+Requires-Dist: argcomplete>=3.6.3
+Requires-Dist: faust_cchardet>=2.1.19
+Requires-Dist: ImageHash>=4.3.2
+Requires-Dist: numpy>=2.2.6
+Requires-Dist: opencv_python>=4.11.0.86
+Requires-Dist: overrides>=7.7.0
+Requires-Dist: Pillow>=12.2.0
+Requires-Dist: platformdirs>=4.9.6
+Requires-Dist: py3langid>=0.3.0
+Requires-Dist: pycountry>=24.6.1
+Requires-Dist: pysubs2>=1.8.0
+Requires-Dist: Requests>=2.33.1
+Requires-Dist: scikit_learn>=1.7.2
+Requires-Dist: scikit-image>=0.25.2
+Requires-Dist: scipy>=1.15.3
+Requires-Dist: tqdm>=4.67.1
+Provides-Extra: ocr
+Requires-Dist: pytesseract; extra == "ocr"
+Provides-Extra: test
+Requires-Dist: parameterized>=0.9.0; extra == "test"
+Dynamic: license-file

twotone-1.0.0/README.md

@@ -0,0 +1,115 @@
+## Tools for Batch Video File Manipulations
+
+### Overview
+
+**TwoTone** is a versatile tool with various subtools for batch video file manipulations.
+
+### Installation
+
+```bash
+pip install -e .
+```
+
+This installs TwoTone in editable mode: the `twotone` command appears on PATH, and any code changes take effect immediately without reinstalling.
+
+### Running TwoTone
+
+```bash
+twotone <global options> <tool-name> <tool-specific options>
+```
+
+### Shell Autocompletion (bash)
+
+```bash
+twotone --install-completion
+```
+
+Open a new terminal. Tab-completion now works:
+
+```
+twotone <TAB>           → concatenate, language_fix, melt, merge, ...
+twotone --<TAB>         → --verbose, --quiet, --no-dry-run, ...
+twotone merge --<TAB>   → merge-specific arguments
+```
+
+To remove: `twotone --uninstall-completion`
+
+### Getting Help
+
+To see a list of global options, available tools, and their descriptions:
+
+```bash
+twotone --help
+```
+
+To get help for a specific tool:
+
+```bash
+twotone <tool-name> --help
+```
+
+### Important Notes
+
+Dry Run Mode: all tools run in a dry run mode by default (no files are being modified). Use the -r or --no-dry-run global option to perform actual operation. In live run mode most tools remove their input files after successful processing.<br/>
+It is safe to stop execution with ctrl+c. All tools handle proper signal and will stop as soon as possible.<br/>
+Working Directory: use -w or --working-dir to specify where temporary and working files are created. By default the directory is taken from the operating system's user data location for the application.<br/>
+
+Data Safety: Always back up your data before using any tool, as source files may be deleted during processing.
+
+### Available Tools
+#### Merge Video Files with Subtitles into MKV Files
+
+The merge tool searches for movie and subtitle files and merges them into a single MKV file.
+
+By default, subtitles are added without a language label. You can specify a language with the --language option.
+
+For a full list of options:
+
+```bash
+twotone merge --help
+```
+
+#### Fix Missing Track Languages
+
+The language_fix tool scans MKV files and fills missing language metadata for subtitle tracks by extracting them and running language detection. It can also infer audio track language from the track name (use --audio to enable; heuristic may be inaccurate).
+
+```bash
+twotone language_fix --help
+```
+
+#### Automatic Video transcoding
+
+The transcode tool transcodes videos using the x265 codec.
+
+It takes a video directory as an input and determines the optimal CRF for each video by comparing the original and encoded versions.
+The script aims to achieve a quality level where SSIM ≈ 0.98 (by default).
+
+For a full list of options:
+
+```bash
+twotone transcode --help
+```
+
+#### Movies concatenation
+
+The concatenate tool looks for movie files which seem to be split into a few files (like CD1, CD2 etc) and glues them into one file.
+
+```bash
+twotone concatenate --help
+```
+
+#### Combine Duplicate Videos (melt)
+
+The melt tool scans for duplicate video files and creates a single output using the best quality segments from each copy. Duplicates can be provided manually or taken from a Jellyfin server. Input files are kept intact by default.
+
+```bash
+twotone melt --help
+```
+
+#### Miscellaneous utilities
+
+The utilities tool groups smaller helpers. Currently it provides the *scenes* subtool for extracting frames from a video and saving them into per-scene folders.
+
+```bash
+twotone utilities scenes --help
+```

twotone-1.0.0/pyproject.toml

@@ -0,0 +1,37 @@
+[build-system]
+requires = ["setuptools>=68.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "twotone"
+version = "1.0.0"
+description = "Tools for batch video file manipulations"
+requires-python = ">=3.10"
+dependencies = [
+    "argcomplete>=3.6.3",
+    "faust_cchardet>=2.1.19",
+    "ImageHash>=4.3.2",
+    "numpy>=2.2.6",
+    "opencv_python>=4.11.0.86",
+    "overrides>=7.7.0",
+    "Pillow>=12.2.0",
+    "platformdirs>=4.9.6",
+    "py3langid>=0.3.0",
+    "pycountry>=24.6.1",
+    "pysubs2>=1.8.0",
+    "Requests>=2.33.1",
+    "scikit_learn>=1.7.2",
+    "scikit-image>=0.25.2",
+    "scipy>=1.15.3",
+    "tqdm>=4.67.1",
+]
+
+[project.optional-dependencies]
+ocr = ["pytesseract"]
+test = ["parameterized>=0.9.0"]
+
+[project.scripts]
+twotone = "twotone.twotone:main"
+
+[tool.setuptools.packages.find]
+include = ["twotone*"]

twotone-1.0.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

twotone-1.0.0/twotone/__main__.py

@@ -0,0 +1,5 @@
+
+from .twotone import main
+
+if __name__ == "__main__":
+    main()

twotone-1.0.0/twotone/tools/concatenate.py

@@ -0,0 +1,201 @@
+
+import argparse
+import logging
+import os
+import re
+from dataclasses import dataclass
+from collections import defaultdict
+from overrides import override
+from tqdm import tqdm
+
+from .tool import EmptyPlan, Plan, Tool
+from twotone.tools.utils import generic_utils, process_utils, video_utils, files_utils
+
+
+class Concatenate(generic_utils.InterruptibleProcess):
+    def __init__(self, logger: logging.Logger, working_dir: str):
+        super().__init__()
+
+        self.logger = logger
+        self.working_dir = working_dir
+
+    def analyze(self, path: str, ignore_warnings: bool = False) -> dict[str, list[tuple[str, int]]] | None:
+        self.logger.info(f"Collecting video files from path {path}")
+        video_files = video_utils.collect_video_files(path, self)
+
+        self.logger.info("Finding splitted videos")
+        parts_regex = re.compile("(.*[^0-9a-z]+)(cd\\d+)([^0-9a-z]+.*)", re.IGNORECASE)
+
+        splitted = []
+        for video_file in video_files:
+            if parts_regex.match(video_file):
+                splitted.append(video_file)
+            else:
+                self.logger.debug(f"File {video_file} does not match pattern")
+
+        self.logger.info("Matching videos")
+        matched_videos: dict[str, list[tuple[str, int]]] = defaultdict(list)
+        for video in splitted:
+            match = parts_regex.search(video)
+            if match is None:
+                continue
+
+            path = match.group(1)
+            if path[-1] == os.sep:
+                # movie path is like: /dir/movie/cd1.mp4
+                # repeat last dir name as a base for output video file
+                last_dir_name = os.path.basename(os.path.normpath(path))
+                name_without_part_number = os.path.join(path, last_dir_name) + match.group(3)
+            else:
+                # movie path is like: /dir/movie/movie cd1.mp4.
+                # remove last char before CDXXX as it is most likely space or hyphen and use it as a base for output video file
+                name_without_part_number = path[:-1] + match.group(3)
+
+            # ffmpeg does not support rmvb container, use mp4
+            dir, name, extn = files_utils.split_path(name_without_part_number)
+            if extn.lower() == "rmvb":
+                name_without_part_number = os.path.join(dir, name + ".mkv")
+
+            part = match.group(2)
+            partNo = int(part[2:])                                                                              # drop 'CD'
+            matched_videos[name_without_part_number].append((video, partNo))
+
+        self.logger.info("Processing groups")
+        warnings = False
+        sorted_videos: dict[str, list[tuple[str, int]]] = {}
+        valid_videos: dict[str, list[tuple[str, int]]] = {}
+        for common_name, details in matched_videos.items():
+
+            # sort parts by part number [1]
+            details = sorted(details, key = lambda detail: detail[1])
+            sorted_videos[common_name] = details
+            group_has_warning = False
+
+            # collect all part numbers
+            parts = []
+            for _, partNo in details:
+                parts.append(partNo)
+
+            if len(parts) < 2:
+                self.logger.warning(f"There are less than two parts for video represented under a common name: {common_name}")
+                warnings = True
+                group_has_warning = True
+
+            # expect parts to be numbered from 1 to N
+            for i, value in enumerate(parts):
+                if i + 1 != value:
+                    self.logger.warning(f"There is a mismatch in CD numbers for a group of files represented under a common name: {common_name}")
+                    warnings = True
+                    group_has_warning = True
+
+            if group_has_warning and ignore_warnings:
+                continue
+            valid_videos[common_name] = details
+
+        if warnings and not ignore_warnings:
+            self.logger.error("Fix above warnings and try again")
+            return None
+
+        if ignore_warnings:
+            return valid_videos
+        return sorted_videos
+
+    def perform(self, sorted_videos: dict[str, list[tuple[str, int]]]) -> None:
+        self.logger.info("Starting concatenation")
+        for output, details in tqdm(sorted_videos.items(), desc="Concatenating", unit="movie", **generic_utils.get_tqdm_defaults()):
+            self.check_for_stop()
+
+            input_files = [video for video, _ in details]
+
+            audio_codec = "copy"
+            for input_file in input_files:
+                file_details = video_utils.get_video_data(input_file)
+                audio_streams = file_details.get("audio", [])
+                for audio_stream in audio_streams:
+                    codec = audio_stream.get("codec")
+                    if codec and codec.lower() == "cook":
+                        audio_codec = "aac"
+                        break
+
+            def escape_path(path: str) -> str:
+                return path.replace("'", "'\\''")
+
+            input_file_content = [f"file '{escape_path(input_file)}'" for input_file in input_files]
+            with files_utils.TempFileManager("\n".join(input_file_content), "txt", directory=self.working_dir) as input_file:
+                ffmpeg_args = ["-f", "concat", "-safe", "0", "-i", input_file, "-c:v", "copy", "-c:a", audio_codec, output]
+
+                self.logger.info(f"Concatenating files into {output} file")
+                status = process_utils.start_process("ffmpeg", ffmpeg_args)
+                if status.returncode == 0:
+                    for input_file in input_files:
+                        os.remove(input_file)
+                else:
+                    self.logger.error(f"Problems with concatenation, skipping file {output}")
+                    self.logger.debug(status.stdout)
+                    self.logger.debug(status.stderr)
+                    if os.path.exists(output):
+                        os.remove(output)
+
+
+class ConcatenateTool(Tool):
+    def __init__(self) -> None:
+        super().__init__()
+
+    @override
+    def setup_parser(self, parser: argparse.ArgumentParser):
+        parser.description = (
+            "Concatenate is a tool for concatenating video files splitted into many files into one.\n"
+            "For example if you have movie consisting of two files: movie-cd1.avi and movie-cd2.avi\n"
+            "then 'concatenate' tool will glue them into one file 'movie.avi'.\n"
+            "If your files come with subtitle files, you may want to use 'merge' tool first\n"
+            "to merge video files with corresponding subtitle files.\n"
+            "Otherwise you will end up with one video file and two subtitle files for cd1 and cd2 which will be useless now"
+        )
+        parser.add_argument('videos_path',
+                            nargs=1,
+                            help='Path with videos to concatenate.')
+        parser.add_argument('--ignore-warnings',
+                            action='store_true',
+                            help='Skip videos with warnings and continue with valid groups.')
+
+    @override
+    def analyze(self, args, logger: logging.Logger, working_dir: str) -> Plan:
+        concatenator = Concatenate(logger, working_dir=working_dir)
+        analysis = concatenator.analyze(args.videos_path[0], ignore_warnings=args.ignore_warnings)
+        if analysis is None:
+            return EmptyPlan()
+        return ConcatenatePlan(items=analysis)
+
+    @override
+    def perform(self, args, logger: logging.Logger, working_dir: str, plan: Plan) -> None:
+        if not isinstance(plan, ConcatenatePlan):
+            raise TypeError(f"Expected ConcatenatePlan, got {type(plan).__name__}")
+
+        concatenator = Concatenate(logger, working_dir)
+        concatenator.perform(plan.items)
+
+
+@dataclass
+class ConcatenatePlan:
+    items: dict[str, list[tuple[str, int]]]
+
+    def is_empty(self) -> bool:
+        return not self.items
+
+    def render(self, logger: logging.Logger) -> None:
+        if not self.items:
+            logger.info("No videos to concatenate.")
+            return
+
+        logger.info("Planned concatenations: %d", len(self.items))
+        for output, details in self.items.items():
+            paths = [path for path, _ in details]
+            common_path = os.path.commonpath(paths) if paths else ""
+            if common_path:
+                logger.info("Files from %s:", common_path)
+                for path, _ in details:
+                    logger.info("  %s", os.path.relpath(path, common_path))
+            else:
+                for path, part in details:
+                    logger.info("  part %d: %s", part, path)
+            logger.info("  -> %s", output)