mkzforge 2.0.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (65) hide show
  1. mkzforge-2.0.0/.cursor/rules/project.mdc +93 -0
  2. mkzforge-2.0.0/.github/workflows/publish.yml +32 -0
  3. mkzforge-2.0.0/.gitignore +59 -0
  4. mkzforge-2.0.0/AGENTS.md +132 -0
  5. mkzforge-2.0.0/CLAUDE.md +132 -0
  6. mkzforge-2.0.0/PKG-INFO +245 -0
  7. mkzforge-2.0.0/README.md +215 -0
  8. mkzforge-2.0.0/bin/ffinfo +44 -0
  9. mkzforge-2.0.0/bin/htmlize +38 -0
  10. mkzforge-2.0.0/bin/t2s.py +10 -0
  11. mkzforge-2.0.0/contrib/sysvinit/mkzforge +146 -0
  12. mkzforge-2.0.0/contrib/sysvinit/mkzforge.default +26 -0
  13. mkzforge-2.0.0/doc/README.md +0 -0
  14. mkzforge-2.0.0/doc/WEBSERVER.md +304 -0
  15. mkzforge-2.0.0/doc/architecture.dia +1107 -0
  16. mkzforge-2.0.0/doc/configuration.md +384 -0
  17. mkzforge-2.0.0/doc/whisper.help.txt +71 -0
  18. mkzforge-2.0.0/lib/mkzforge/__init__.py +45 -0
  19. mkzforge-2.0.0/lib/mkzforge/__main__.py +5 -0
  20. mkzforge-2.0.0/lib/mkzforge/_version.py +1 -0
  21. mkzforge-2.0.0/lib/mkzforge/cli/__init__.py +413 -0
  22. mkzforge-2.0.0/lib/mkzforge/cli/build.py +27 -0
  23. mkzforge-2.0.0/lib/mkzforge/cli/new.py +30 -0
  24. mkzforge-2.0.0/lib/mkzforge/cli/normalize.py +124 -0
  25. mkzforge-2.0.0/lib/mkzforge/cli/publish.py +51 -0
  26. mkzforge-2.0.0/lib/mkzforge/cli/web.py +90 -0
  27. mkzforge-2.0.0/lib/mkzforge/const.py +27 -0
  28. mkzforge-2.0.0/lib/mkzforge/filter_complex.py +189 -0
  29. mkzforge-2.0.0/lib/mkzforge/genimg.py +210 -0
  30. mkzforge-2.0.0/lib/mkzforge/grive.py +253 -0
  31. mkzforge-2.0.0/lib/mkzforge/i18n.py +218 -0
  32. mkzforge-2.0.0/lib/mkzforge/metadata.py +81 -0
  33. mkzforge-2.0.0/lib/mkzforge/notify.py +184 -0
  34. mkzforge-2.0.0/lib/mkzforge/schema.json +452 -0
  35. mkzforge-2.0.0/lib/mkzforge/subtitles.py +172 -0
  36. mkzforge-2.0.0/lib/mkzforge/types.py +27 -0
  37. mkzforge-2.0.0/lib/mkzforge/utils.py +253 -0
  38. mkzforge-2.0.0/lib/mkzforge/videos.py +566 -0
  39. mkzforge-2.0.0/lib/mkzforge/webserv.py +411 -0
  40. mkzforge-2.0.0/lib/mkzforge.egg-info/PKG-INFO +245 -0
  41. mkzforge-2.0.0/lib/mkzforge.egg-info/SOURCES.txt +63 -0
  42. mkzforge-2.0.0/lib/mkzforge.egg-info/dependency_links.txt +1 -0
  43. mkzforge-2.0.0/lib/mkzforge.egg-info/entry_points.txt +2 -0
  44. mkzforge-2.0.0/lib/mkzforge.egg-info/requires.txt +19 -0
  45. mkzforge-2.0.0/lib/mkzforge.egg-info/top_level.txt +1 -0
  46. mkzforge-2.0.0/pyproject.toml +66 -0
  47. mkzforge-2.0.0/setup.cfg +4 -0
  48. mkzforge-2.0.0/tests/fixtures/configs/base.yml +5 -0
  49. mkzforge-2.0.0/tests/fixtures/configs/extended-filter-complex.yml +17 -0
  50. mkzforge-2.0.0/tests/fixtures/configs/extended.yml +20 -0
  51. mkzforge-2.0.0/tests/fixtures/configs/metadata.yml +13 -0
  52. mkzforge-2.0.0/tests/fixtures/configs/simple-filter-complex.yml +8 -0
  53. mkzforge-2.0.0/tests/mkzforgefunc/__init__.py +1 -0
  54. mkzforge-2.0.0/tests/mkzforgefunc/vertical.py +359 -0
  55. mkzforge-2.0.0/tests/mkzforgeunit/__init__.py +0 -0
  56. mkzforge-2.0.0/tests/mkzforgeunit/build.py +140 -0
  57. mkzforge-2.0.0/tests/mkzforgeunit/config.py +53 -0
  58. mkzforge-2.0.0/tests/mkzforgeunit/gpu_lock.py +250 -0
  59. mkzforge-2.0.0/tests/mkzforgeunit/webserv.py +221 -0
  60. mkzforge-2.0.0/tests/runtests.py +16 -0
  61. mkzforge-2.0.0/web/app.js +243 -0
  62. mkzforge-2.0.0/web/index.html +133 -0
  63. mkzforge-2.0.0/web/project-detail.html +103 -0
  64. mkzforge-2.0.0/web/projects.html +78 -0
  65. mkzforge-2.0.0/web/style.css +520 -0
@@ -0,0 +1,93 @@
1
+ # Python, FFmpeg, OpenAI Whisper, YAML
2
+
3
+ **You are an expert Python developer specializing in video/audio processing automation, with deep experience using FFmpeg, OpenAI Whisper, YAML-based configuration systems, and automation scripting.**
4
+
5
+ ## File Editing Best Practices
6
+
7
+ * **Complete File Replacements**: When creating new files or when a file is in a broken state, replace the entire file content rather than attempting partial edits.
8
+ * **Docstring and String Literals**: Always provide complete, syntactically correct blocks with explicit opening and closing quotes. Avoid partial replacements of string content.
9
+ * **Error Recovery**: If linter errors appear after an edit, don't try to "fix" the file with additional edits. Request a complete file replacement with correct syntax.
10
+ * **Token Efficiency**: While complete replacements use more tokens upfront, they often result in lower total cost by eliminating multiple failed attempts.
11
+ * **Clear Context**: Always specify exact locations for changes and whether you're replacing entire files or specific sections.
12
+ * **Syntax-Sensitive Elements**: For imports, class definitions, docstrings, and other syntax-critical code, prefer complete file replacements over partial edits.
13
+
14
+ ## Key Principles
15
+
16
+ * Write concise, efficient Python with clear, practical examples.
17
+ * Prioritize clarity, modularity, and automation-friendly design.
18
+ * Favor function composition and reusable utilities over over-engineering with classes.
19
+ * Use `subprocess` for reliable FFmpeg/CLI calls with robust error handling.
20
+ * Follow PEP 8 and functional programming best practices.
21
+
22
+ ## FFmpeg Integration
23
+
24
+ * Use `subprocess` or `ffmpeg-python` for invoking FFmpeg commands.
25
+ * Validate FFmpeg command-line arguments rigorously.
26
+ * Capture and parse FFmpeg output when needed for metadata.
27
+ * Handle transcoding, filtering, audio processing, and stream mapping with explicit command composition.
28
+ * Prefer raw CLI commands for flexibility when `ffmpeg-python` cannot express certain options.
29
+
30
+ ## OpenAI Whisper Usage
31
+
32
+ * Use the OpenAI Whisper Python API or CLI wrapper effectively.
33
+ * Automate transcription workflows with batching and error recovery.
34
+ * Handle large audio files via chunking or segmentation if necessary.
35
+ * Validate model selection (tiny, base, small, medium, large) based on use-case.
36
+ * Implement automatic language detection where possible.
37
+ * Integrate Whisper output into post-processing pipelines (e.g., subtitle generation).
38
+
39
+ ## YAML-Driven Workflow Design
40
+
41
+ * Use `PyYAML` for safe YAML parsing and schema enforcement.
42
+ * Design declarative YAML schemas for defining video generation pipelines.
43
+ * Allow YAML to configure:
44
+
45
+ * Input sources
46
+ * FFmpeg operations (e.g., filters, trims, overlays)
47
+ * Whisper transcription options
48
+ * Output formats and destinations
49
+ * Validate YAML structures strictly before execution.
50
+
51
+ ## Automation Scripting Best Practices
52
+
53
+ * Build reusable utilities for command construction and execution.
54
+ * Modularize pipeline stages (ingest, process, transcribe, output).
55
+ * Implement dry-run modes for command preview before execution.
56
+ * Handle exceptions explicitly and log all subprocess outputs.
57
+ * Support CLI arguments for overrides of YAML-defined parameters.
58
+
59
+ ## Error Handling & Debugging
60
+
61
+ * Wrap all subprocess calls with try-except blocks.
62
+ * Decode subprocess stdout/stderr for meaningful error messages.
63
+ * Use verbose logging during debugging and silent modes in production.
64
+ * Implement retry logic for transient errors in long automation jobs.
65
+
66
+ ## Performance & Resource Management
67
+
68
+ * Optimize FFmpeg pipelines with filter chains to minimize disk I/O.
69
+ * Use temporary files or pipes when chaining processes.
70
+ * Manage large file handling with streaming where possible.
71
+ * Monitor system resource usage (CPU, disk I/O) during heavy processing tasks.
72
+
73
+ ## Dependencies
74
+
75
+ * Python 3.11+
76
+ * `PyYAML` (for YAML parsing)
77
+ * `python-ffmpeg` (optional)
78
+ * `subprocess` (standard)
79
+ * OpenAI Whisper Python package
80
+
81
+ ## Key Conventions
82
+
83
+ 1. All configs and runtime parameters should be YAML-driven.
84
+ 2. Separate script concerns: configuration, command execution, post-processing.
85
+ 3. Always support verbose/debug output for troubleshooting.
86
+ 4. Document example YAML files alongside code.
87
+
88
+ ## Refer to
89
+
90
+ * Official FFmpeg documentation: [https://ffmpeg.org/documentation.html](https://ffmpeg.org/documentation.html)
91
+ * OpenAI Whisper GitHub: [https://github.com/openai/whisper](https://github.com/openai/whisper)
92
+ * PyYAML documentation: [https://pyyaml.org/](https://pyyaml.org/)
93
+ * Python `subprocess` documentation: [https://docs.python.org/3/library/subprocess.html](https://docs.python.org/3/library/subprocess.html)
@@ -0,0 +1,32 @@
1
+ name: Publish to PyPI
2
+
3
+ on:
4
+ push:
5
+ tags:
6
+ - 'v*'
7
+
8
+ jobs:
9
+ publish:
10
+ runs-on: ubuntu-latest
11
+
12
+ steps:
13
+ - uses: actions/checkout@v4
14
+ with:
15
+ fetch-depth: 0
16
+
17
+ - name: Set up Python
18
+ uses: actions/setup-python@v5
19
+ with:
20
+ python-version: '3.11'
21
+
22
+ - name: Install uv and twine
23
+ run: pip install uv twine
24
+
25
+ - name: Build package
26
+ run: uv build
27
+
28
+ - name: Publish to PyPI
29
+ env:
30
+ TWINE_USERNAME: __token__
31
+ TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
32
+ run: twine upload dist/*
@@ -0,0 +1,59 @@
1
+ # Compressed archives
2
+ *.tgz
3
+ *.tar
4
+ *.gz
5
+ *.bz2
6
+ *.lz4
7
+
8
+ # Media files
9
+ *.webm
10
+ *.mp4
11
+ *.mp3
12
+ *.m4a
13
+ *.mkv
14
+ *.mov
15
+ *.aac
16
+ *.ts
17
+ *.ac3
18
+ *.wav
19
+
20
+ # Images
21
+ *.png
22
+ *.jpg
23
+ *.jpeg
24
+ *.gif
25
+ *.xcf
26
+ *.ttf
27
+ *.svg
28
+
29
+ # Documents
30
+ *.docx
31
+ *.pdf
32
+ *.xlsx
33
+
34
+ # System Files
35
+ ## Windows
36
+ desktop.ini
37
+ ## OSX
38
+ .DS_Store
39
+ ## IDE's
40
+ .nbproject/
41
+ .idea/
42
+
43
+ # Python cache
44
+ __pycache__/
45
+ build/
46
+ *.egg-info
47
+ .eggs
48
+ dist/
49
+ lib/mkzforge/_version.py
50
+
51
+ # Makefile's are generated, so no need to keep them under VCS.
52
+ Makefile
53
+
54
+ # Experiments with Natron until I'm confident it will be used.
55
+ *.ntp
56
+ *.ntp.~*~
57
+
58
+ .venv
59
+ uv.lock
@@ -0,0 +1,132 @@
1
+ # CLAUDE.md
2
+
3
+ This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
4
+
5
+ ## Project Overview
6
+
7
+ MKZ Forge is a Python video processing automation tool that wraps FFmpeg's `filter_complex` into a
8
+ YAML-driven pipeline. It handles the full workflow: MP4→MKV normalization, silence removal,
9
+ Whisper subtitle generation, multi-language translation, LLM metadata generation, Gemini thumbnail
10
+ generation, and publishing.
11
+
12
+ ## Commands
13
+
14
+ ### Install & Setup
15
+
16
+ ```bash
17
+ uv pip install -e . # standard editable install
18
+ ```
19
+
20
+ ### Running
21
+
22
+ ```bash
23
+ mkzforge --help
24
+ mkzforge new [path] # Create new project
25
+ mkzforge normalize # Normalize videos (compress, silence removal, subtitles)
26
+ mkzforge build [output] # Build final video from mkzforge.yml
27
+ mkzforge gensubs [video] # Generate subtitles only
28
+ mkzforge metadata # Generate LLM title/description
29
+ mkzforge genimage # Generate Gemini thumbnail
30
+ mkzforge publish [file] # Publish to SFTP/YouTube/TikTok
31
+ mkzforge serve [--workspace /path] [--http-port 9091] # Start web interface
32
+ ```
33
+
34
+ ### Testing
35
+
36
+ ```bash
37
+ uv run pytest # Run all tests, including the virtualenv.
38
+ ```
39
+
40
+ Tests are split into `tests/mkzforgeunit/` (unit) and `tests/mkzforgefunc/`
41
+ (functional/integration). Fixtures live in `tests/fixtures/configs/`.
42
+
43
+ ## Architecture
44
+
45
+ ### Pipeline Flow
46
+
47
+ 1. `mkzforge normalize` → converts MP4→MKV (H.265 crf=28), removes silence via FFmpeg `silencedetect`,
48
+ runs Whisper for subtitles, translates via Argos Translate, calls LLM for metadata
49
+ 2. `mkzforge build` → reads `mkzforge.yml`, constructs FFmpeg filter graph via `filter_complex.py`
50
+ DSL, produces final MP4
51
+ 3. `mkzforge publish` → uploads via SFTP (Fabric), with hooks for YouTube/TikTok
52
+
53
+ ### Key Modules
54
+
55
+ | Module | Purpose |
56
+ |--------|---------|
57
+ | `cli/__init__.py` | Argument parsing, action dispatch, config merging |
58
+ | `videos.py` | MP4→MKV, silence detection, video compilation |
59
+ | `subtitles.py` | Whisper integration, SRT parsing/writing |
60
+ | `i18n.py` | Argos Translate integration, subtitle re-timing |
61
+ | `filter_complex.py` | FFmpeg filter graph DSL abstraction |
62
+ | `metadata.py` | LangChain LLM calls for titles/descriptions |
63
+ | `genimg.py` | Google Gemini image generation, SVG thumbnails |
64
+ | `webserv.py` | CherryPy REST API + background workers |
65
+ | `notify.py` | AWS SNS singleton |
66
+ | `utils.py` | Config loading, GPU VRAM detection |
67
+ | `types.py` | Enums: `Devices`, `Action`, `WhisperTask` |
68
+ | `const.py` | Supported languages, API key constants |
69
+
70
+ ### Config Loading Order (merged, last wins)
71
+
72
+ 1. `/etc/mkzforge/config.yml`
73
+ 2. `~/.config/mkzforge/config.yml`
74
+ 3. `./mkzforge.yml` (project-specific)
75
+ 4. CLI arguments
76
+
77
+ ### Project Directory Structure
78
+
79
+ ```plain
80
+ my-project/
81
+ ├── mkzforge.yml # YAML config describing build pipeline
82
+ ├── resources/ # Input MP4/MKV files
83
+ └── build/ # Generated outputs (MKV, SRT, MP4, PNG)
84
+ ```
85
+
86
+ ### `mkzforge.yml` Schema
87
+
88
+ ```yaml
89
+ videos:
90
+ - input: [list of files/sources]
91
+ output: output.mp4
92
+ attributes: [subs, no-video, no-audio, vsync, no-publish, thumbnail]
93
+ languages: [en, es, fr]
94
+ filter_complex: [ffmpeg filter directives]
95
+ metadata:
96
+ title: "..."
97
+ description: "..."
98
+ map:
99
+ video: "[v]"
100
+ audio: "[a]"
101
+ subs: 0
102
+ ```
103
+
104
+ Full schema documented in `lib/mkzforge/schema.json` and `doc/configuration.md`.
105
+
106
+ ### Whisper Model Auto-Selection (by GPU VRAM)
107
+
108
+ - ≥10GB → `large-v3`; 8–10GB → `large-v2`; 6–8GB → `medium`; 3–6GB → `small`; <3GB → `base`; CPU
109
+ fallback → `small`
110
+
111
+ ### GPU Locking
112
+
113
+ File-based lock prevents concurrent GPU usage across multiple `mkzforge` processes.
114
+
115
+ ## Key Environment Variables
116
+
117
+ | Variable | Default | Purpose |
118
+ |----------|---------|---------|
119
+ | `LANGUAGE` | `en` | Base transcription language |
120
+ | `LOG_LEVEL` | `INFO` | Logging verbosity |
121
+ | `WHISPER_MODEL` | auto | Override Whisper model selection |
122
+ | `LLM_MODEL` | `gpt-oss:20b` | LLM model name |
123
+ | `LLM_PROVIDER` | `ollama` | LLM provider |
124
+ | `GOOGLE_API_KEY` | — | Gemini API key |
125
+ | `GEMINI_IMAGE_MODEL` | `gemini-2.5-flash-image` | Gemini model |
126
+ | `HTTP_PORT` | `9091` | Web server port |
127
+ | `OMP_NUM_THREADS` | `nproc` | CPU thread count |
128
+
129
+ ## Versioning
130
+
131
+ Uses `setuptools-scm` — version is derived from git tags (`v*` format). Version is written to
132
+ `lib/mkzforge/_version.py` automatically.
@@ -0,0 +1,132 @@
1
+ # CLAUDE.md
2
+
3
+ This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
4
+
5
+ ## Project Overview
6
+
7
+ MKZ Forge is a Python video processing automation tool that wraps FFmpeg's `filter_complex` into a
8
+ YAML-driven pipeline. It handles the full workflow: MP4→MKV normalization, silence removal,
9
+ Whisper subtitle generation, multi-language translation, LLM metadata generation, Gemini thumbnail
10
+ generation, and publishing.
11
+
12
+ ## Commands
13
+
14
+ ### Install & Setup
15
+
16
+ ```bash
17
+ uv pip install -e . # standard editable install
18
+ ```
19
+
20
+ ### Running
21
+
22
+ ```bash
23
+ mkzforge --help
24
+ mkzforge new [path] # Create new project
25
+ mkzforge normalize # Normalize videos (compress, silence removal, subtitles)
26
+ mkzforge build [output] # Build final video from mkzforge.yml
27
+ mkzforge gensubs [video] # Generate subtitles only
28
+ mkzforge metadata # Generate LLM title/description
29
+ mkzforge genimage # Generate Gemini thumbnail
30
+ mkzforge publish [file] # Publish to SFTP/YouTube/TikTok
31
+ mkzforge serve [--workspace /path] [--http-port 9091] # Start web interface
32
+ ```
33
+
34
+ ### Testing
35
+
36
+ ```bash
37
+ uv run pytest # Run all tests, including the virtualenv.
38
+ ```
39
+
40
+ Tests are split into `tests/mkzforgeunit/` (unit) and `tests/mkzforgefunc/`
41
+ (functional/integration). Fixtures live in `tests/fixtures/configs/`.
42
+
43
+ ## Architecture
44
+
45
+ ### Pipeline Flow
46
+
47
+ 1. `mkzforge normalize` → converts MP4→MKV (H.265 crf=28), removes silence via FFmpeg `silencedetect`,
48
+ runs Whisper for subtitles, translates via Argos Translate, calls LLM for metadata
49
+ 2. `mkzforge build` → reads `mkzforge.yml`, constructs FFmpeg filter graph via `filter_complex.py`
50
+ DSL, produces final MP4
51
+ 3. `mkzforge publish` → uploads via SFTP (Fabric), with hooks for YouTube/TikTok
52
+
53
+ ### Key Modules
54
+
55
+ | Module | Purpose |
56
+ |--------|---------|
57
+ | `cli/__init__.py` | Argument parsing, action dispatch, config merging |
58
+ | `videos.py` | MP4→MKV, silence detection, video compilation |
59
+ | `subtitles.py` | Whisper integration, SRT parsing/writing |
60
+ | `i18n.py` | Argos Translate integration, subtitle re-timing |
61
+ | `filter_complex.py` | FFmpeg filter graph DSL abstraction |
62
+ | `metadata.py` | LangChain LLM calls for titles/descriptions |
63
+ | `genimg.py` | Google Gemini image generation, SVG thumbnails |
64
+ | `webserv.py` | CherryPy REST API + background workers |
65
+ | `notify.py` | AWS SNS singleton |
66
+ | `utils.py` | Config loading, GPU VRAM detection |
67
+ | `types.py` | Enums: `Devices`, `Action`, `WhisperTask` |
68
+ | `const.py` | Supported languages, API key constants |
69
+
70
+ ### Config Loading Order (merged, last wins)
71
+
72
+ 1. `/etc/mkzforge/config.yml`
73
+ 2. `~/.config/mkzforge/config.yml`
74
+ 3. `./mkzforge.yml` (project-specific)
75
+ 4. CLI arguments
76
+
77
+ ### Project Directory Structure
78
+
79
+ ```plain
80
+ my-project/
81
+ ├── mkzforge.yml # YAML config describing build pipeline
82
+ ├── resources/ # Input MP4/MKV files
83
+ └── build/ # Generated outputs (MKV, SRT, MP4, PNG)
84
+ ```
85
+
86
+ ### `mkzforge.yml` Schema
87
+
88
+ ```yaml
89
+ videos:
90
+ - input: [list of files/sources]
91
+ output: output.mp4
92
+ attributes: [subs, no-video, no-audio, vsync, no-publish, thumbnail]
93
+ languages: [en, es, fr]
94
+ filter_complex: [ffmpeg filter directives]
95
+ metadata:
96
+ title: "..."
97
+ description: "..."
98
+ map:
99
+ video: "[v]"
100
+ audio: "[a]"
101
+ subs: 0
102
+ ```
103
+
104
+ Full schema documented in `lib/mkzforge/schema.json` and `doc/configuration.md`.
105
+
106
+ ### Whisper Model Auto-Selection (by GPU VRAM)
107
+
108
+ - ≥10GB → `large-v3`; 8–10GB → `large-v2`; 6–8GB → `medium`; 3–6GB → `small`; <3GB → `base`; CPU
109
+ fallback → `small`
110
+
111
+ ### GPU Locking
112
+
113
+ File-based lock prevents concurrent GPU usage across multiple `mkzforge` processes.
114
+
115
+ ## Key Environment Variables
116
+
117
+ | Variable | Default | Purpose |
118
+ |----------|---------|---------|
119
+ | `LANGUAGE` | `en` | Base transcription language |
120
+ | `LOG_LEVEL` | `INFO` | Logging verbosity |
121
+ | `WHISPER_MODEL` | auto | Override Whisper model selection |
122
+ | `LLM_MODEL` | `gpt-oss:20b` | LLM model name |
123
+ | `LLM_PROVIDER` | `ollama` | LLM provider |
124
+ | `GOOGLE_API_KEY` | — | Gemini API key |
125
+ | `GEMINI_IMAGE_MODEL` | `gemini-2.5-flash-image` | Gemini model |
126
+ | `HTTP_PORT` | `9091` | Web server port |
127
+ | `OMP_NUM_THREADS` | `nproc` | CPU thread count |
128
+
129
+ ## Versioning
130
+
131
+ Uses `setuptools-scm` — version is derived from git tags (`v*` format). Version is written to
132
+ `lib/mkzforge/_version.py` automatically.
@@ -0,0 +1,245 @@
1
+ Metadata-Version: 2.4
2
+ Name: mkzforge
3
+ Version: 2.0.0
4
+ Summary: Scripts and tools to ease the processing of videos for Social Media platforms.
5
+ Author-email: Markizano Draconus <markizano@markizano.net>
6
+ Project-URL: Homepage, https://mkzforge.markizano.net
7
+ Project-URL: Repository, https://github.com/markizano/mkzforge
8
+ Project-URL: Issues, https://github.com/markizano/mkzforge/issues
9
+ Requires-Python: >=3.11
10
+ Description-Content-Type: text/markdown
11
+ Requires-Dist: argostranslate>=1.11.0
12
+ Requires-Dist: boto3>=1.42.59
13
+ Requires-Dist: cherrypy>=18.10.0
14
+ Requires-Dist: fabric==3.2.2
15
+ Requires-Dist: google-api-python-client>=2.193.0
16
+ Requires-Dist: google-auth-httplib2>=0.3.0
17
+ Requires-Dist: google-auth-oauthlib>=1.3.0
18
+ Requires-Dist: google-genai>=1.48.0
19
+ Requires-Dist: jsonschema>=4.25.1
20
+ Requires-Dist: kizano==1.0.6
21
+ Requires-Dist: langchain>=1.0.5
22
+ Requires-Dist: langchain-ollama>=1.0.0
23
+ Requires-Dist: numba>=0.62.1
24
+ Requires-Dist: openai>=2.7.2
25
+ Requires-Dist: openai-whisper==20250625
26
+ Requires-Dist: pillow>=10.0.0
27
+ Requires-Dist: python-ffmpeg==2.0.12
28
+ Requires-Dist: pyyaml>=6.0.1
29
+ Requires-Dist: requests>=2.32.5
30
+
31
+ # MKZ Forge
32
+
33
+ > Markizano's Forge for video production!
34
+
35
+ This is a Python project I use and maintain to automate various aspects of my video editing
36
+ process to publish on video supporting platforms like TikTok, YouTube, LinkedIn and Substack.
37
+
38
+ FFMPEG is a great video editor by itself, but it's difficult to manage that incredible description
39
+ library it has.
40
+
41
+ This project handles the following process for me:
42
+
43
+ * **Convert MP4 files to MKV** files to compress for disk space.
44
+ * **Cut Silence** from videos to remove the "uhm"s and "uuh"s.
45
+ * **Combine videos** so multiple uploads can result in a single artifact to publish.
46
+ * **Generate subtitles** using OpenAI's `whisper` libraries for hardsubs/captions.
47
+ * **Generate Metadata** for title and description based on the content of the video.
48
+ * **Generate Thumbnail** using Google's GenAI Image generation API.
49
+ * **Pristine Video Production** for that final product you see on video platforms posted by me.
50
+
51
+ In a single upload (or multi-file upload), I can quickly generate videos that already have
52
+ captions hardcoded into them, metadata attached so platforms can easily and quickly categorize
53
+ my videos for publishing (if they support that kind of thing).
54
+
55
+ With this project, I hope to make it easier for program-oriented folks like me to have a
56
+ descriptive file that converts your videos for you instead of having to remember complex filter
57
+ graphs. Simply describe what you want the video to do and this app will help you in executing those
58
+ changes.
59
+
60
+ This is a very context-specific application that is catered to my needs. I never expected it to
61
+ become popular, but if it does, here's to all who may contribute to this project or find it useful
62
+ in their own video editing process endeavours.
63
+
64
+ ## Usage
65
+
66
+ When recording from phone or cam, it's difficult to manage all the media and content that comes in
67
+ without paying for crazy software to get it done. With this open-source product, you can simply
68
+ `pip install mkzforge` and go!
69
+
70
+ This project honours configuration defined in a system-wide configuration file in `/etc/mkzforge/config.yml`,
71
+ user-specific configuration defined in `~/.config/mkzforge/config.yml`, with the configurations merged
72
+ together in that order as defined.
73
+
74
+ To create a new project, let's use this:
75
+
76
+ ```bash
77
+ mkzforge new
78
+ ```
79
+
80
+ This will render a new project with the following directory structure:
81
+
82
+ ```none
83
+ .
84
+ ├── build/
85
+ ├── readme.md
86
+ ├── resources/
87
+ └── mkzforge.yml
88
+ ```
89
+
90
+ You can drop your MP4 files from your devices into the `./resources/` directory.
91
+
92
+ Next, we can run
93
+
94
+ ```bash
95
+ mkzforge normalize
96
+ ```
97
+
98
+ to refresh the YAML file that is the configuration driving the
99
+ changes we will be performing here.
100
+
101
+ This may take a moment as ffmpeg converts your MP4 format videos into MKV format under high
102
+ compression as lossless as possible. This will reduce the amount of disk that is consumed by the
103
+ videos recorded and saved raw from devices. Subtitles will also be automatically generated from
104
+ the video files using Whisper! The system automatically detects your GPU VRAM and selects the
105
+ best Whisper model that fits (preferring `large-v3` for high-end GPUs).
106
+
107
+ **Concurrent Processing:** If you run multiple mkzforge instances simultaneously, they will
108
+ automatically queue for GPU access using a file-based lock to prevent OOM (Out Of Memory) errors.
109
+ Each instance waits its turn instead of competing for GPU resources.
110
+
111
+ You can suppress auto-subtitle generation with the `--no-subtitles` argument.
112
+
113
+ Once your videos have been compressed and subtitles generated for them, you will have artifacts
114
+ available in the `./build/` directory as well.
115
+
116
+ INFO: See doc/configuration.md for more information about `mkzforge.yml` configuration.
117
+
118
+ You can use this:
119
+
120
+ ```bash
121
+ mkzforge gensubs [path-to-file.mkv]
122
+ ```
123
+
124
+ This will generate subtitles in `build/path-to-file.${LANG}.srt` for any video file you give this
125
+ command. This is not necessary if you used `mkzforge refresh` to generate subs from a video
126
+ resource. I used this to get access to the sub-generation functionality this app provided with this
127
+ command and so I exposed that function to this command here. You can elect to do more with it after
128
+ that.
129
+
130
+ Once your videos have been compressed and your configuration updated, you will see the YAML
131
+ has been updated with the new video. If you have a preferred name for it, you should rename
132
+ the file prior to running `mkzforge refresh`.
133
+
134
+ Observe that subtitles will also be generated as a result of this update. To avoid this, you can
135
+ use `--no-subtitles` when executing `mkzforge refresh --no-auto-subtitles` and it will go a bit faster.
136
+
137
+ You can update the YAML configuration to have it execute a number of filters and stream the videos
138
+ together into a final cut that can be used for social media sites and such.
139
+
140
+ The top-level `videos` is an array which will contain the set of video descriptions you will use
141
+ to describe how you want transformations done on those videos.
142
+
143
+ To learn more about the mkzforge.yml file, see docs/mkzforge.md
144
+
145
+ Once you have your transformations written out, you can use this:
146
+
147
+ ```bash
148
+ mkzforge build build/myvideo.mp4
149
+ ```
150
+
151
+ This will build your video. If you omit any arguments, it'll attempt to build all videos in your
152
+ project and ensure they are ready to go!
153
+
154
+ This project takes some of the rough edges off of the `filter_complex` argument in ffmpeg.
155
+
156
+ This started off as a simple script to try and automate some of the rough edges of my process
157
+ when recording content and publishing to the platforms.
158
+
159
+ ## Multi-Language Subtitle Support
160
+
161
+ mkzforge now supports automatic translation of subtitles to multiple languages using Argos Translate!
162
+
163
+ ### Quick Start
164
+
165
+ 1. **Configure languages in your `mkzforge.yml`:**
166
+
167
+ ```yaml
168
+ mkzforge:
169
+ language: en # Base language for Whisper transcription
170
+ languages: # Translate to these languages
171
+ - en # English (from Whisper)
172
+ - es # Spanish (translated)
173
+ - fr # French (translated)
174
+ - de # German (translated)
175
+ ```
176
+
177
+ 2. **Run refresh to generate and translate subtitles:**
178
+
179
+ ```bash
180
+ mkzforge refresh
181
+ ```
182
+
183
+ This will:
184
+
185
+ * Generate English subtitles using Whisper
186
+ * Automatically translate to Spanish, French, and German
187
+ * Preserve timing information from the original subtitles
188
+ * Create separate SRT files for each language in `build/`
189
+
190
+ 3. **Build your video with all subtitle tracks:**
191
+
192
+ ```bash
193
+ mkzforge build
194
+ ```
195
+
196
+ The final video will contain all subtitle tracks, properly mapped and labeled.
197
+
198
+ ### How Translation Works
199
+
200
+ **Context-Aware Translation**: The entire transcript is translated as one document to preserve
201
+ meaning, intent, and context. This produces much better results than translating line-by-line.
202
+
203
+ **Timing Preservation**: After translation, the system intelligently splits the translated text to
204
+ match the original subtitle timing, distributing words proportionally across subtitle entries.
205
+
206
+ **Automatic Package Management**: Argos Translate language packages are downloaded and installed
207
+ automatically on first use.
208
+
209
+ ### Example Output
210
+
211
+ After running `mkzforge refresh` with multi-language support:
212
+
213
+ ```plain
214
+ build/
215
+ ├── my-video.mkv # Processed video
216
+ ├── my-video.txt # Full English transcript
217
+ ├── my-video.en.srt # English subtitles (Whisper)
218
+ ├── my-video.es.srt # Spanish translation
219
+ ├── my-video.fr.srt # French translation
220
+ ├── my-video.de.srt # German translation
221
+ └── my-video.mp4 # Final video with all subtitle tracks
222
+ ```
223
+
224
+ ### Supported Languages
225
+
226
+ Any language pair supported by Argos Translate, including:
227
+
228
+ * Spanish (es), French (fr), German (de), Portuguese (pt)
229
+ * Italian (it), Russian (ru), Chinese (zh), Japanese (ja)
230
+ * Arabic (ar), Hindi (hi), Korean (ko)
231
+ * And many more!
232
+
233
+ For detailed documentation, see [CHANGES.md](CHANGES.md).
234
+
235
+ ## @FutureFeature
236
+
237
+ Features I'd like to add to this include:
238
+
239
+ * `mkzforge publish` to publish to your configured social media platforms
240
+ * I want to support YouTube, TikTok, and Mastadon.
241
+ * Right now, only an SFTP endpoint is supported (uses [Fabric](https://www.fabfile.org/)).
242
+ * Stream specifier error clarification: It would be nice to know that a stream is disconnected
243
+ right there in vscode or some editor of sorts. I hope a project like this can make that more
244
+ easily accesible as a potential by simulating the results of the configuration in ffmpeg's
245
+ `filter_complex` parameter.