mkzforge 2.0.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- mkzforge-2.0.0/.cursor/rules/project.mdc +93 -0
- mkzforge-2.0.0/.github/workflows/publish.yml +32 -0
- mkzforge-2.0.0/.gitignore +59 -0
- mkzforge-2.0.0/AGENTS.md +132 -0
- mkzforge-2.0.0/CLAUDE.md +132 -0
- mkzforge-2.0.0/PKG-INFO +245 -0
- mkzforge-2.0.0/README.md +215 -0
- mkzforge-2.0.0/bin/ffinfo +44 -0
- mkzforge-2.0.0/bin/htmlize +38 -0
- mkzforge-2.0.0/bin/t2s.py +10 -0
- mkzforge-2.0.0/contrib/sysvinit/mkzforge +146 -0
- mkzforge-2.0.0/contrib/sysvinit/mkzforge.default +26 -0
- mkzforge-2.0.0/doc/README.md +0 -0
- mkzforge-2.0.0/doc/WEBSERVER.md +304 -0
- mkzforge-2.0.0/doc/architecture.dia +1107 -0
- mkzforge-2.0.0/doc/configuration.md +384 -0
- mkzforge-2.0.0/doc/whisper.help.txt +71 -0
- mkzforge-2.0.0/lib/mkzforge/__init__.py +45 -0
- mkzforge-2.0.0/lib/mkzforge/__main__.py +5 -0
- mkzforge-2.0.0/lib/mkzforge/_version.py +1 -0
- mkzforge-2.0.0/lib/mkzforge/cli/__init__.py +413 -0
- mkzforge-2.0.0/lib/mkzforge/cli/build.py +27 -0
- mkzforge-2.0.0/lib/mkzforge/cli/new.py +30 -0
- mkzforge-2.0.0/lib/mkzforge/cli/normalize.py +124 -0
- mkzforge-2.0.0/lib/mkzforge/cli/publish.py +51 -0
- mkzforge-2.0.0/lib/mkzforge/cli/web.py +90 -0
- mkzforge-2.0.0/lib/mkzforge/const.py +27 -0
- mkzforge-2.0.0/lib/mkzforge/filter_complex.py +189 -0
- mkzforge-2.0.0/lib/mkzforge/genimg.py +210 -0
- mkzforge-2.0.0/lib/mkzforge/grive.py +253 -0
- mkzforge-2.0.0/lib/mkzforge/i18n.py +218 -0
- mkzforge-2.0.0/lib/mkzforge/metadata.py +81 -0
- mkzforge-2.0.0/lib/mkzforge/notify.py +184 -0
- mkzforge-2.0.0/lib/mkzforge/schema.json +452 -0
- mkzforge-2.0.0/lib/mkzforge/subtitles.py +172 -0
- mkzforge-2.0.0/lib/mkzforge/types.py +27 -0
- mkzforge-2.0.0/lib/mkzforge/utils.py +253 -0
- mkzforge-2.0.0/lib/mkzforge/videos.py +566 -0
- mkzforge-2.0.0/lib/mkzforge/webserv.py +411 -0
- mkzforge-2.0.0/lib/mkzforge.egg-info/PKG-INFO +245 -0
- mkzforge-2.0.0/lib/mkzforge.egg-info/SOURCES.txt +63 -0
- mkzforge-2.0.0/lib/mkzforge.egg-info/dependency_links.txt +1 -0
- mkzforge-2.0.0/lib/mkzforge.egg-info/entry_points.txt +2 -0
- mkzforge-2.0.0/lib/mkzforge.egg-info/requires.txt +19 -0
- mkzforge-2.0.0/lib/mkzforge.egg-info/top_level.txt +1 -0
- mkzforge-2.0.0/pyproject.toml +66 -0
- mkzforge-2.0.0/setup.cfg +4 -0
- mkzforge-2.0.0/tests/fixtures/configs/base.yml +5 -0
- mkzforge-2.0.0/tests/fixtures/configs/extended-filter-complex.yml +17 -0
- mkzforge-2.0.0/tests/fixtures/configs/extended.yml +20 -0
- mkzforge-2.0.0/tests/fixtures/configs/metadata.yml +13 -0
- mkzforge-2.0.0/tests/fixtures/configs/simple-filter-complex.yml +8 -0
- mkzforge-2.0.0/tests/mkzforgefunc/__init__.py +1 -0
- mkzforge-2.0.0/tests/mkzforgefunc/vertical.py +359 -0
- mkzforge-2.0.0/tests/mkzforgeunit/__init__.py +0 -0
- mkzforge-2.0.0/tests/mkzforgeunit/build.py +140 -0
- mkzforge-2.0.0/tests/mkzforgeunit/config.py +53 -0
- mkzforge-2.0.0/tests/mkzforgeunit/gpu_lock.py +250 -0
- mkzforge-2.0.0/tests/mkzforgeunit/webserv.py +221 -0
- mkzforge-2.0.0/tests/runtests.py +16 -0
- mkzforge-2.0.0/web/app.js +243 -0
- mkzforge-2.0.0/web/index.html +133 -0
- mkzforge-2.0.0/web/project-detail.html +103 -0
- mkzforge-2.0.0/web/projects.html +78 -0
- mkzforge-2.0.0/web/style.css +520 -0
|
@@ -0,0 +1,93 @@
|
|
|
1
|
+
# Python, FFmpeg, OpenAI Whisper, YAML
|
|
2
|
+
|
|
3
|
+
**You are an expert Python developer specializing in video/audio processing automation, with deep experience using FFmpeg, OpenAI Whisper, YAML-based configuration systems, and automation scripting.**
|
|
4
|
+
|
|
5
|
+
## File Editing Best Practices
|
|
6
|
+
|
|
7
|
+
* **Complete File Replacements**: When creating new files or when a file is in a broken state, replace the entire file content rather than attempting partial edits.
|
|
8
|
+
* **Docstring and String Literals**: Always provide complete, syntactically correct blocks with explicit opening and closing quotes. Avoid partial replacements of string content.
|
|
9
|
+
* **Error Recovery**: If linter errors appear after an edit, don't try to "fix" the file with additional edits. Request a complete file replacement with correct syntax.
|
|
10
|
+
* **Token Efficiency**: While complete replacements use more tokens upfront, they often result in lower total cost by eliminating multiple failed attempts.
|
|
11
|
+
* **Clear Context**: Always specify exact locations for changes and whether you're replacing entire files or specific sections.
|
|
12
|
+
* **Syntax-Sensitive Elements**: For imports, class definitions, docstrings, and other syntax-critical code, prefer complete file replacements over partial edits.
|
|
13
|
+
|
|
14
|
+
## Key Principles
|
|
15
|
+
|
|
16
|
+
* Write concise, efficient Python with clear, practical examples.
|
|
17
|
+
* Prioritize clarity, modularity, and automation-friendly design.
|
|
18
|
+
* Favor function composition and reusable utilities over over-engineering with classes.
|
|
19
|
+
* Use `subprocess` for reliable FFmpeg/CLI calls with robust error handling.
|
|
20
|
+
* Follow PEP 8 and functional programming best practices.
|
|
21
|
+
|
|
22
|
+
## FFmpeg Integration
|
|
23
|
+
|
|
24
|
+
* Use `subprocess` or `ffmpeg-python` for invoking FFmpeg commands.
|
|
25
|
+
* Validate FFmpeg command-line arguments rigorously.
|
|
26
|
+
* Capture and parse FFmpeg output when needed for metadata.
|
|
27
|
+
* Handle transcoding, filtering, audio processing, and stream mapping with explicit command composition.
|
|
28
|
+
* Prefer raw CLI commands for flexibility when `ffmpeg-python` cannot express certain options.
|
|
29
|
+
|
|
30
|
+
## OpenAI Whisper Usage
|
|
31
|
+
|
|
32
|
+
* Use the OpenAI Whisper Python API or CLI wrapper effectively.
|
|
33
|
+
* Automate transcription workflows with batching and error recovery.
|
|
34
|
+
* Handle large audio files via chunking or segmentation if necessary.
|
|
35
|
+
* Validate model selection (tiny, base, small, medium, large) based on use-case.
|
|
36
|
+
* Implement automatic language detection where possible.
|
|
37
|
+
* Integrate Whisper output into post-processing pipelines (e.g., subtitle generation).
|
|
38
|
+
|
|
39
|
+
## YAML-Driven Workflow Design
|
|
40
|
+
|
|
41
|
+
* Use `PyYAML` for safe YAML parsing and schema enforcement.
|
|
42
|
+
* Design declarative YAML schemas for defining video generation pipelines.
|
|
43
|
+
* Allow YAML to configure:
|
|
44
|
+
|
|
45
|
+
* Input sources
|
|
46
|
+
* FFmpeg operations (e.g., filters, trims, overlays)
|
|
47
|
+
* Whisper transcription options
|
|
48
|
+
* Output formats and destinations
|
|
49
|
+
* Validate YAML structures strictly before execution.
|
|
50
|
+
|
|
51
|
+
## Automation Scripting Best Practices
|
|
52
|
+
|
|
53
|
+
* Build reusable utilities for command construction and execution.
|
|
54
|
+
* Modularize pipeline stages (ingest, process, transcribe, output).
|
|
55
|
+
* Implement dry-run modes for command preview before execution.
|
|
56
|
+
* Handle exceptions explicitly and log all subprocess outputs.
|
|
57
|
+
* Support CLI arguments for overrides of YAML-defined parameters.
|
|
58
|
+
|
|
59
|
+
## Error Handling & Debugging
|
|
60
|
+
|
|
61
|
+
* Wrap all subprocess calls with try-except blocks.
|
|
62
|
+
* Decode subprocess stdout/stderr for meaningful error messages.
|
|
63
|
+
* Use verbose logging during debugging and silent modes in production.
|
|
64
|
+
* Implement retry logic for transient errors in long automation jobs.
|
|
65
|
+
|
|
66
|
+
## Performance & Resource Management
|
|
67
|
+
|
|
68
|
+
* Optimize FFmpeg pipelines with filter chains to minimize disk I/O.
|
|
69
|
+
* Use temporary files or pipes when chaining processes.
|
|
70
|
+
* Manage large file handling with streaming where possible.
|
|
71
|
+
* Monitor system resource usage (CPU, disk I/O) during heavy processing tasks.
|
|
72
|
+
|
|
73
|
+
## Dependencies
|
|
74
|
+
|
|
75
|
+
* Python 3.11+
|
|
76
|
+
* `PyYAML` (for YAML parsing)
|
|
77
|
+
* `python-ffmpeg` (optional)
|
|
78
|
+
* `subprocess` (standard)
|
|
79
|
+
* OpenAI Whisper Python package
|
|
80
|
+
|
|
81
|
+
## Key Conventions
|
|
82
|
+
|
|
83
|
+
1. All configs and runtime parameters should be YAML-driven.
|
|
84
|
+
2. Separate script concerns: configuration, command execution, post-processing.
|
|
85
|
+
3. Always support verbose/debug output for troubleshooting.
|
|
86
|
+
4. Document example YAML files alongside code.
|
|
87
|
+
|
|
88
|
+
## Refer to
|
|
89
|
+
|
|
90
|
+
* Official FFmpeg documentation: [https://ffmpeg.org/documentation.html](https://ffmpeg.org/documentation.html)
|
|
91
|
+
* OpenAI Whisper GitHub: [https://github.com/openai/whisper](https://github.com/openai/whisper)
|
|
92
|
+
* PyYAML documentation: [https://pyyaml.org/](https://pyyaml.org/)
|
|
93
|
+
* Python `subprocess` documentation: [https://docs.python.org/3/library/subprocess.html](https://docs.python.org/3/library/subprocess.html)
|
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
name: Publish to PyPI
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
push:
|
|
5
|
+
tags:
|
|
6
|
+
- 'v*'
|
|
7
|
+
|
|
8
|
+
jobs:
|
|
9
|
+
publish:
|
|
10
|
+
runs-on: ubuntu-latest
|
|
11
|
+
|
|
12
|
+
steps:
|
|
13
|
+
- uses: actions/checkout@v4
|
|
14
|
+
with:
|
|
15
|
+
fetch-depth: 0
|
|
16
|
+
|
|
17
|
+
- name: Set up Python
|
|
18
|
+
uses: actions/setup-python@v5
|
|
19
|
+
with:
|
|
20
|
+
python-version: '3.11'
|
|
21
|
+
|
|
22
|
+
- name: Install uv and twine
|
|
23
|
+
run: pip install uv twine
|
|
24
|
+
|
|
25
|
+
- name: Build package
|
|
26
|
+
run: uv build
|
|
27
|
+
|
|
28
|
+
- name: Publish to PyPI
|
|
29
|
+
env:
|
|
30
|
+
TWINE_USERNAME: __token__
|
|
31
|
+
TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
|
|
32
|
+
run: twine upload dist/*
|
|
@@ -0,0 +1,59 @@
|
|
|
1
|
+
# Compressed archives
|
|
2
|
+
*.tgz
|
|
3
|
+
*.tar
|
|
4
|
+
*.gz
|
|
5
|
+
*.bz2
|
|
6
|
+
*.lz4
|
|
7
|
+
|
|
8
|
+
# Media files
|
|
9
|
+
*.webm
|
|
10
|
+
*.mp4
|
|
11
|
+
*.mp3
|
|
12
|
+
*.m4a
|
|
13
|
+
*.mkv
|
|
14
|
+
*.mov
|
|
15
|
+
*.aac
|
|
16
|
+
*.ts
|
|
17
|
+
*.ac3
|
|
18
|
+
*.wav
|
|
19
|
+
|
|
20
|
+
# Images
|
|
21
|
+
*.png
|
|
22
|
+
*.jpg
|
|
23
|
+
*.jpeg
|
|
24
|
+
*.gif
|
|
25
|
+
*.xcf
|
|
26
|
+
*.ttf
|
|
27
|
+
*.svg
|
|
28
|
+
|
|
29
|
+
# Documents
|
|
30
|
+
*.docx
|
|
31
|
+
*.pdf
|
|
32
|
+
*.xlsx
|
|
33
|
+
|
|
34
|
+
# System Files
|
|
35
|
+
## Windows
|
|
36
|
+
desktop.ini
|
|
37
|
+
## OSX
|
|
38
|
+
.DS_Store
|
|
39
|
+
## IDE's
|
|
40
|
+
.nbproject/
|
|
41
|
+
.idea/
|
|
42
|
+
|
|
43
|
+
# Python cache
|
|
44
|
+
__pycache__/
|
|
45
|
+
build/
|
|
46
|
+
*.egg-info
|
|
47
|
+
.eggs
|
|
48
|
+
dist/
|
|
49
|
+
lib/mkzforge/_version.py
|
|
50
|
+
|
|
51
|
+
# Makefile's are generated, so no need to keep them under VCS.
|
|
52
|
+
Makefile
|
|
53
|
+
|
|
54
|
+
# Experiments with Natron until I'm confident it will be used.
|
|
55
|
+
*.ntp
|
|
56
|
+
*.ntp.~*~
|
|
57
|
+
|
|
58
|
+
.venv
|
|
59
|
+
uv.lock
|
mkzforge-2.0.0/AGENTS.md
ADDED
|
@@ -0,0 +1,132 @@
|
|
|
1
|
+
# CLAUDE.md
|
|
2
|
+
|
|
3
|
+
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
|
4
|
+
|
|
5
|
+
## Project Overview
|
|
6
|
+
|
|
7
|
+
MKZ Forge is a Python video processing automation tool that wraps FFmpeg's `filter_complex` into a
|
|
8
|
+
YAML-driven pipeline. It handles the full workflow: MP4→MKV normalization, silence removal,
|
|
9
|
+
Whisper subtitle generation, multi-language translation, LLM metadata generation, Gemini thumbnail
|
|
10
|
+
generation, and publishing.
|
|
11
|
+
|
|
12
|
+
## Commands
|
|
13
|
+
|
|
14
|
+
### Install & Setup
|
|
15
|
+
|
|
16
|
+
```bash
|
|
17
|
+
uv pip install -e . # standard editable install
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
### Running
|
|
21
|
+
|
|
22
|
+
```bash
|
|
23
|
+
mkzforge --help
|
|
24
|
+
mkzforge new [path] # Create new project
|
|
25
|
+
mkzforge normalize # Normalize videos (compress, silence removal, subtitles)
|
|
26
|
+
mkzforge build [output] # Build final video from mkzforge.yml
|
|
27
|
+
mkzforge gensubs [video] # Generate subtitles only
|
|
28
|
+
mkzforge metadata # Generate LLM title/description
|
|
29
|
+
mkzforge genimage # Generate Gemini thumbnail
|
|
30
|
+
mkzforge publish [file] # Publish to SFTP/YouTube/TikTok
|
|
31
|
+
mkzforge serve [--workspace /path] [--http-port 9091] # Start web interface
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
### Testing
|
|
35
|
+
|
|
36
|
+
```bash
|
|
37
|
+
uv run pytest # Run all tests, including the virtualenv.
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
Tests are split into `tests/mkzforgeunit/` (unit) and `tests/mkzforgefunc/`
|
|
41
|
+
(functional/integration). Fixtures live in `tests/fixtures/configs/`.
|
|
42
|
+
|
|
43
|
+
## Architecture
|
|
44
|
+
|
|
45
|
+
### Pipeline Flow
|
|
46
|
+
|
|
47
|
+
1. `mkzforge normalize` → converts MP4→MKV (H.265 crf=28), removes silence via FFmpeg `silencedetect`,
|
|
48
|
+
runs Whisper for subtitles, translates via Argos Translate, calls LLM for metadata
|
|
49
|
+
2. `mkzforge build` → reads `mkzforge.yml`, constructs FFmpeg filter graph via `filter_complex.py`
|
|
50
|
+
DSL, produces final MP4
|
|
51
|
+
3. `mkzforge publish` → uploads via SFTP (Fabric), with hooks for YouTube/TikTok
|
|
52
|
+
|
|
53
|
+
### Key Modules
|
|
54
|
+
|
|
55
|
+
| Module | Purpose |
|
|
56
|
+
|--------|---------|
|
|
57
|
+
| `cli/__init__.py` | Argument parsing, action dispatch, config merging |
|
|
58
|
+
| `videos.py` | MP4→MKV, silence detection, video compilation |
|
|
59
|
+
| `subtitles.py` | Whisper integration, SRT parsing/writing |
|
|
60
|
+
| `i18n.py` | Argos Translate integration, subtitle re-timing |
|
|
61
|
+
| `filter_complex.py` | FFmpeg filter graph DSL abstraction |
|
|
62
|
+
| `metadata.py` | LangChain LLM calls for titles/descriptions |
|
|
63
|
+
| `genimg.py` | Google Gemini image generation, SVG thumbnails |
|
|
64
|
+
| `webserv.py` | CherryPy REST API + background workers |
|
|
65
|
+
| `notify.py` | AWS SNS singleton |
|
|
66
|
+
| `utils.py` | Config loading, GPU VRAM detection |
|
|
67
|
+
| `types.py` | Enums: `Devices`, `Action`, `WhisperTask` |
|
|
68
|
+
| `const.py` | Supported languages, API key constants |
|
|
69
|
+
|
|
70
|
+
### Config Loading Order (merged, last wins)
|
|
71
|
+
|
|
72
|
+
1. `/etc/mkzforge/config.yml`
|
|
73
|
+
2. `~/.config/mkzforge/config.yml`
|
|
74
|
+
3. `./mkzforge.yml` (project-specific)
|
|
75
|
+
4. CLI arguments
|
|
76
|
+
|
|
77
|
+
### Project Directory Structure
|
|
78
|
+
|
|
79
|
+
```plain
|
|
80
|
+
my-project/
|
|
81
|
+
├── mkzforge.yml # YAML config describing build pipeline
|
|
82
|
+
├── resources/ # Input MP4/MKV files
|
|
83
|
+
└── build/ # Generated outputs (MKV, SRT, MP4, PNG)
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
### `mkzforge.yml` Schema
|
|
87
|
+
|
|
88
|
+
```yaml
|
|
89
|
+
videos:
|
|
90
|
+
- input: [list of files/sources]
|
|
91
|
+
output: output.mp4
|
|
92
|
+
attributes: [subs, no-video, no-audio, vsync, no-publish, thumbnail]
|
|
93
|
+
languages: [en, es, fr]
|
|
94
|
+
filter_complex: [ffmpeg filter directives]
|
|
95
|
+
metadata:
|
|
96
|
+
title: "..."
|
|
97
|
+
description: "..."
|
|
98
|
+
map:
|
|
99
|
+
video: "[v]"
|
|
100
|
+
audio: "[a]"
|
|
101
|
+
subs: 0
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
Full schema documented in `lib/mkzforge/schema.json` and `doc/configuration.md`.
|
|
105
|
+
|
|
106
|
+
### Whisper Model Auto-Selection (by GPU VRAM)
|
|
107
|
+
|
|
108
|
+
- ≥10GB → `large-v3`; 8–10GB → `large-v2`; 6–8GB → `medium`; 3–6GB → `small`; <3GB → `base`; CPU
|
|
109
|
+
fallback → `small`
|
|
110
|
+
|
|
111
|
+
### GPU Locking
|
|
112
|
+
|
|
113
|
+
File-based lock prevents concurrent GPU usage across multiple `mkzforge` processes.
|
|
114
|
+
|
|
115
|
+
## Key Environment Variables
|
|
116
|
+
|
|
117
|
+
| Variable | Default | Purpose |
|
|
118
|
+
|----------|---------|---------|
|
|
119
|
+
| `LANGUAGE` | `en` | Base transcription language |
|
|
120
|
+
| `LOG_LEVEL` | `INFO` | Logging verbosity |
|
|
121
|
+
| `WHISPER_MODEL` | auto | Override Whisper model selection |
|
|
122
|
+
| `LLM_MODEL` | `gpt-oss:20b` | LLM model name |
|
|
123
|
+
| `LLM_PROVIDER` | `ollama` | LLM provider |
|
|
124
|
+
| `GOOGLE_API_KEY` | — | Gemini API key |
|
|
125
|
+
| `GEMINI_IMAGE_MODEL` | `gemini-2.5-flash-image` | Gemini model |
|
|
126
|
+
| `HTTP_PORT` | `9091` | Web server port |
|
|
127
|
+
| `OMP_NUM_THREADS` | `nproc` | CPU thread count |
|
|
128
|
+
|
|
129
|
+
## Versioning
|
|
130
|
+
|
|
131
|
+
Uses `setuptools-scm` — version is derived from git tags (`v*` format). Version is written to
|
|
132
|
+
`lib/mkzforge/_version.py` automatically.
|
mkzforge-2.0.0/CLAUDE.md
ADDED
|
@@ -0,0 +1,132 @@
|
|
|
1
|
+
# CLAUDE.md
|
|
2
|
+
|
|
3
|
+
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
|
4
|
+
|
|
5
|
+
## Project Overview
|
|
6
|
+
|
|
7
|
+
MKZ Forge is a Python video processing automation tool that wraps FFmpeg's `filter_complex` into a
|
|
8
|
+
YAML-driven pipeline. It handles the full workflow: MP4→MKV normalization, silence removal,
|
|
9
|
+
Whisper subtitle generation, multi-language translation, LLM metadata generation, Gemini thumbnail
|
|
10
|
+
generation, and publishing.
|
|
11
|
+
|
|
12
|
+
## Commands
|
|
13
|
+
|
|
14
|
+
### Install & Setup
|
|
15
|
+
|
|
16
|
+
```bash
|
|
17
|
+
uv pip install -e . # standard editable install
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
### Running
|
|
21
|
+
|
|
22
|
+
```bash
|
|
23
|
+
mkzforge --help
|
|
24
|
+
mkzforge new [path] # Create new project
|
|
25
|
+
mkzforge normalize # Normalize videos (compress, silence removal, subtitles)
|
|
26
|
+
mkzforge build [output] # Build final video from mkzforge.yml
|
|
27
|
+
mkzforge gensubs [video] # Generate subtitles only
|
|
28
|
+
mkzforge metadata # Generate LLM title/description
|
|
29
|
+
mkzforge genimage # Generate Gemini thumbnail
|
|
30
|
+
mkzforge publish [file] # Publish to SFTP/YouTube/TikTok
|
|
31
|
+
mkzforge serve [--workspace /path] [--http-port 9091] # Start web interface
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
### Testing
|
|
35
|
+
|
|
36
|
+
```bash
|
|
37
|
+
uv run pytest # Run all tests, including the virtualenv.
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
Tests are split into `tests/mkzforgeunit/` (unit) and `tests/mkzforgefunc/`
|
|
41
|
+
(functional/integration). Fixtures live in `tests/fixtures/configs/`.
|
|
42
|
+
|
|
43
|
+
## Architecture
|
|
44
|
+
|
|
45
|
+
### Pipeline Flow
|
|
46
|
+
|
|
47
|
+
1. `mkzforge normalize` → converts MP4→MKV (H.265 crf=28), removes silence via FFmpeg `silencedetect`,
|
|
48
|
+
runs Whisper for subtitles, translates via Argos Translate, calls LLM for metadata
|
|
49
|
+
2. `mkzforge build` → reads `mkzforge.yml`, constructs FFmpeg filter graph via `filter_complex.py`
|
|
50
|
+
DSL, produces final MP4
|
|
51
|
+
3. `mkzforge publish` → uploads via SFTP (Fabric), with hooks for YouTube/TikTok
|
|
52
|
+
|
|
53
|
+
### Key Modules
|
|
54
|
+
|
|
55
|
+
| Module | Purpose |
|
|
56
|
+
|--------|---------|
|
|
57
|
+
| `cli/__init__.py` | Argument parsing, action dispatch, config merging |
|
|
58
|
+
| `videos.py` | MP4→MKV, silence detection, video compilation |
|
|
59
|
+
| `subtitles.py` | Whisper integration, SRT parsing/writing |
|
|
60
|
+
| `i18n.py` | Argos Translate integration, subtitle re-timing |
|
|
61
|
+
| `filter_complex.py` | FFmpeg filter graph DSL abstraction |
|
|
62
|
+
| `metadata.py` | LangChain LLM calls for titles/descriptions |
|
|
63
|
+
| `genimg.py` | Google Gemini image generation, SVG thumbnails |
|
|
64
|
+
| `webserv.py` | CherryPy REST API + background workers |
|
|
65
|
+
| `notify.py` | AWS SNS singleton |
|
|
66
|
+
| `utils.py` | Config loading, GPU VRAM detection |
|
|
67
|
+
| `types.py` | Enums: `Devices`, `Action`, `WhisperTask` |
|
|
68
|
+
| `const.py` | Supported languages, API key constants |
|
|
69
|
+
|
|
70
|
+
### Config Loading Order (merged, last wins)
|
|
71
|
+
|
|
72
|
+
1. `/etc/mkzforge/config.yml`
|
|
73
|
+
2. `~/.config/mkzforge/config.yml`
|
|
74
|
+
3. `./mkzforge.yml` (project-specific)
|
|
75
|
+
4. CLI arguments
|
|
76
|
+
|
|
77
|
+
### Project Directory Structure
|
|
78
|
+
|
|
79
|
+
```plain
|
|
80
|
+
my-project/
|
|
81
|
+
├── mkzforge.yml # YAML config describing build pipeline
|
|
82
|
+
├── resources/ # Input MP4/MKV files
|
|
83
|
+
└── build/ # Generated outputs (MKV, SRT, MP4, PNG)
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
### `mkzforge.yml` Schema
|
|
87
|
+
|
|
88
|
+
```yaml
|
|
89
|
+
videos:
|
|
90
|
+
- input: [list of files/sources]
|
|
91
|
+
output: output.mp4
|
|
92
|
+
attributes: [subs, no-video, no-audio, vsync, no-publish, thumbnail]
|
|
93
|
+
languages: [en, es, fr]
|
|
94
|
+
filter_complex: [ffmpeg filter directives]
|
|
95
|
+
metadata:
|
|
96
|
+
title: "..."
|
|
97
|
+
description: "..."
|
|
98
|
+
map:
|
|
99
|
+
video: "[v]"
|
|
100
|
+
audio: "[a]"
|
|
101
|
+
subs: 0
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
Full schema documented in `lib/mkzforge/schema.json` and `doc/configuration.md`.
|
|
105
|
+
|
|
106
|
+
### Whisper Model Auto-Selection (by GPU VRAM)
|
|
107
|
+
|
|
108
|
+
- ≥10GB → `large-v3`; 8–10GB → `large-v2`; 6–8GB → `medium`; 3–6GB → `small`; <3GB → `base`; CPU
|
|
109
|
+
fallback → `small`
|
|
110
|
+
|
|
111
|
+
### GPU Locking
|
|
112
|
+
|
|
113
|
+
File-based lock prevents concurrent GPU usage across multiple `mkzforge` processes.
|
|
114
|
+
|
|
115
|
+
## Key Environment Variables
|
|
116
|
+
|
|
117
|
+
| Variable | Default | Purpose |
|
|
118
|
+
|----------|---------|---------|
|
|
119
|
+
| `LANGUAGE` | `en` | Base transcription language |
|
|
120
|
+
| `LOG_LEVEL` | `INFO` | Logging verbosity |
|
|
121
|
+
| `WHISPER_MODEL` | auto | Override Whisper model selection |
|
|
122
|
+
| `LLM_MODEL` | `gpt-oss:20b` | LLM model name |
|
|
123
|
+
| `LLM_PROVIDER` | `ollama` | LLM provider |
|
|
124
|
+
| `GOOGLE_API_KEY` | — | Gemini API key |
|
|
125
|
+
| `GEMINI_IMAGE_MODEL` | `gemini-2.5-flash-image` | Gemini model |
|
|
126
|
+
| `HTTP_PORT` | `9091` | Web server port |
|
|
127
|
+
| `OMP_NUM_THREADS` | `nproc` | CPU thread count |
|
|
128
|
+
|
|
129
|
+
## Versioning
|
|
130
|
+
|
|
131
|
+
Uses `setuptools-scm` — version is derived from git tags (`v*` format). Version is written to
|
|
132
|
+
`lib/mkzforge/_version.py` automatically.
|
mkzforge-2.0.0/PKG-INFO
ADDED
|
@@ -0,0 +1,245 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: mkzforge
|
|
3
|
+
Version: 2.0.0
|
|
4
|
+
Summary: Scripts and tools to ease the processing of videos for Social Media platforms.
|
|
5
|
+
Author-email: Markizano Draconus <markizano@markizano.net>
|
|
6
|
+
Project-URL: Homepage, https://mkzforge.markizano.net
|
|
7
|
+
Project-URL: Repository, https://github.com/markizano/mkzforge
|
|
8
|
+
Project-URL: Issues, https://github.com/markizano/mkzforge/issues
|
|
9
|
+
Requires-Python: >=3.11
|
|
10
|
+
Description-Content-Type: text/markdown
|
|
11
|
+
Requires-Dist: argostranslate>=1.11.0
|
|
12
|
+
Requires-Dist: boto3>=1.42.59
|
|
13
|
+
Requires-Dist: cherrypy>=18.10.0
|
|
14
|
+
Requires-Dist: fabric==3.2.2
|
|
15
|
+
Requires-Dist: google-api-python-client>=2.193.0
|
|
16
|
+
Requires-Dist: google-auth-httplib2>=0.3.0
|
|
17
|
+
Requires-Dist: google-auth-oauthlib>=1.3.0
|
|
18
|
+
Requires-Dist: google-genai>=1.48.0
|
|
19
|
+
Requires-Dist: jsonschema>=4.25.1
|
|
20
|
+
Requires-Dist: kizano==1.0.6
|
|
21
|
+
Requires-Dist: langchain>=1.0.5
|
|
22
|
+
Requires-Dist: langchain-ollama>=1.0.0
|
|
23
|
+
Requires-Dist: numba>=0.62.1
|
|
24
|
+
Requires-Dist: openai>=2.7.2
|
|
25
|
+
Requires-Dist: openai-whisper==20250625
|
|
26
|
+
Requires-Dist: pillow>=10.0.0
|
|
27
|
+
Requires-Dist: python-ffmpeg==2.0.12
|
|
28
|
+
Requires-Dist: pyyaml>=6.0.1
|
|
29
|
+
Requires-Dist: requests>=2.32.5
|
|
30
|
+
|
|
31
|
+
# MKZ Forge
|
|
32
|
+
|
|
33
|
+
> Markizano's Forge for video production!
|
|
34
|
+
|
|
35
|
+
This is a Python project I use and maintain to automate various aspects of my video editing
|
|
36
|
+
process to publish on video supporting platforms like TikTok, YouTube, LinkedIn and Substack.
|
|
37
|
+
|
|
38
|
+
FFMPEG is a great video editor by itself, but it's difficult to manage that incredible description
|
|
39
|
+
library it has.
|
|
40
|
+
|
|
41
|
+
This project handles the following process for me:
|
|
42
|
+
|
|
43
|
+
* **Convert MP4 files to MKV** files to compress for disk space.
|
|
44
|
+
* **Cut Silence** from videos to remove the "uhm"s and "uuh"s.
|
|
45
|
+
* **Combine videos** so multiple uploads can result in a single artifact to publish.
|
|
46
|
+
* **Generate subtitles** using OpenAI's `whisper` libraries for hardsubs/captions.
|
|
47
|
+
* **Generate Metadata** for title and description based on the content of the video.
|
|
48
|
+
* **Generate Thumbnail** using Google's GenAI Image generation API.
|
|
49
|
+
* **Pristine Video Production** for that final product you see on video platforms posted by me.
|
|
50
|
+
|
|
51
|
+
In a single upload (or multi-file upload), I can quickly generate videos that already have
|
|
52
|
+
captions hardcoded into them, metadata attached so platforms can easily and quickly categorize
|
|
53
|
+
my videos for publishing (if they support that kind of thing).
|
|
54
|
+
|
|
55
|
+
With this project, I hope to make it easier for program-oriented folks like me to have a
|
|
56
|
+
descriptive file that converts your videos for you instead of having to remember complex filter
|
|
57
|
+
graphs. Simply describe what you want the video to do and this app will help you in executing those
|
|
58
|
+
changes.
|
|
59
|
+
|
|
60
|
+
This is a very context-specific application that is catered to my needs. I never expected it to
|
|
61
|
+
become popular, but if it does, here's to all who may contribute to this project or find it useful
|
|
62
|
+
in their own video editing process endeavours.
|
|
63
|
+
|
|
64
|
+
## Usage
|
|
65
|
+
|
|
66
|
+
When recording from phone or cam, it's difficult to manage all the media and content that comes in
|
|
67
|
+
without paying for crazy software to get it done. With this open-source product, you can simply
|
|
68
|
+
`pip install mkzforge` and go!
|
|
69
|
+
|
|
70
|
+
This project honours configuration defined in a system-wide configuration file in `/etc/mkzforge/config.yml`,
|
|
71
|
+
user-specific configuration defined in `~/.config/mkzforge/config.yml`, with the configurations merged
|
|
72
|
+
together in that order as defined.
|
|
73
|
+
|
|
74
|
+
To create a new project, let's use this:
|
|
75
|
+
|
|
76
|
+
```bash
|
|
77
|
+
mkzforge new
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
This will render a new project with the following directory structure:
|
|
81
|
+
|
|
82
|
+
```none
|
|
83
|
+
.
|
|
84
|
+
├── build/
|
|
85
|
+
├── readme.md
|
|
86
|
+
├── resources/
|
|
87
|
+
└── mkzforge.yml
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
You can drop your MP4 files from your devices into the `./resources/` directory.
|
|
91
|
+
|
|
92
|
+
Next, we can run
|
|
93
|
+
|
|
94
|
+
```bash
|
|
95
|
+
mkzforge normalize
|
|
96
|
+
```
|
|
97
|
+
|
|
98
|
+
to refresh the YAML file that is the configuration driving the
|
|
99
|
+
changes we will be performing here.
|
|
100
|
+
|
|
101
|
+
This may take a moment as ffmpeg converts your MP4 format videos into MKV format under high
|
|
102
|
+
compression as lossless as possible. This will reduce the amount of disk that is consumed by the
|
|
103
|
+
videos recorded and saved raw from devices. Subtitles will also be automatically generated from
|
|
104
|
+
the video files using Whisper! The system automatically detects your GPU VRAM and selects the
|
|
105
|
+
best Whisper model that fits (preferring `large-v3` for high-end GPUs).
|
|
106
|
+
|
|
107
|
+
**Concurrent Processing:** If you run multiple mkzforge instances simultaneously, they will
|
|
108
|
+
automatically queue for GPU access using a file-based lock to prevent OOM (Out Of Memory) errors.
|
|
109
|
+
Each instance waits its turn instead of competing for GPU resources.
|
|
110
|
+
|
|
111
|
+
You can suppress auto-subtitle generation with the `--no-subtitles` argument.
|
|
112
|
+
|
|
113
|
+
Once your videos have been compressed and subtitles generated for them, you will have artifacts
|
|
114
|
+
available in the `./build/` directory as well.
|
|
115
|
+
|
|
116
|
+
INFO: See doc/configuration.md for more information about `mkzforge.yml` configuration.
|
|
117
|
+
|
|
118
|
+
You can use this:
|
|
119
|
+
|
|
120
|
+
```bash
|
|
121
|
+
mkzforge gensubs [path-to-file.mkv]
|
|
122
|
+
```
|
|
123
|
+
|
|
124
|
+
This will generate subtitles in `build/path-to-file.${LANG}.srt` for any video file you give this
|
|
125
|
+
command. This is not necessary if you used `mkzforge refresh` to generate subs from a video
|
|
126
|
+
resource. I used this to get access to the sub-generation functionality this app provided with this
|
|
127
|
+
command and so I exposed that function to this command here. You can elect to do more with it after
|
|
128
|
+
that.
|
|
129
|
+
|
|
130
|
+
Once your videos have been compressed and your configuration updated, you will see the YAML
|
|
131
|
+
has been updated with the new video. If you have a preferred name for it, you should rename
|
|
132
|
+
the file prior to running `mkzforge refresh`.
|
|
133
|
+
|
|
134
|
+
Observe that subtitles will also be generated as a result of this update. To avoid this, you can
|
|
135
|
+
use `--no-subtitles` when executing `mkzforge refresh --no-auto-subtitles` and it will go a bit faster.
|
|
136
|
+
|
|
137
|
+
You can update the YAML configuration to have it execute a number of filters and stream the videos
|
|
138
|
+
together into a final cut that can be used for social media sites and such.
|
|
139
|
+
|
|
140
|
+
The top-level `videos` is an array which will contain the set of video descriptions you will use
|
|
141
|
+
to describe how you want transformations done on those videos.
|
|
142
|
+
|
|
143
|
+
To learn more about the mkzforge.yml file, see docs/mkzforge.md
|
|
144
|
+
|
|
145
|
+
Once you have your transformations written out, you can use this:
|
|
146
|
+
|
|
147
|
+
```bash
|
|
148
|
+
mkzforge build build/myvideo.mp4
|
|
149
|
+
```
|
|
150
|
+
|
|
151
|
+
This will build your video. If you omit any arguments, it'll attempt to build all videos in your
|
|
152
|
+
project and ensure they are ready to go!
|
|
153
|
+
|
|
154
|
+
This project takes some of the rough edges off of the `filter_complex` argument in ffmpeg.
|
|
155
|
+
|
|
156
|
+
This started off as a simple script to try and automate some of the rough edges of my process
|
|
157
|
+
when recording content and publishing to the platforms.
|
|
158
|
+
|
|
159
|
+
## Multi-Language Subtitle Support
|
|
160
|
+
|
|
161
|
+
mkzforge now supports automatic translation of subtitles to multiple languages using Argos Translate!
|
|
162
|
+
|
|
163
|
+
### Quick Start
|
|
164
|
+
|
|
165
|
+
1. **Configure languages in your `mkzforge.yml`:**
|
|
166
|
+
|
|
167
|
+
```yaml
|
|
168
|
+
mkzforge:
|
|
169
|
+
language: en # Base language for Whisper transcription
|
|
170
|
+
languages: # Translate to these languages
|
|
171
|
+
- en # English (from Whisper)
|
|
172
|
+
- es # Spanish (translated)
|
|
173
|
+
- fr # French (translated)
|
|
174
|
+
- de # German (translated)
|
|
175
|
+
```
|
|
176
|
+
|
|
177
|
+
2. **Run refresh to generate and translate subtitles:**
|
|
178
|
+
|
|
179
|
+
```bash
|
|
180
|
+
mkzforge refresh
|
|
181
|
+
```
|
|
182
|
+
|
|
183
|
+
This will:
|
|
184
|
+
|
|
185
|
+
* Generate English subtitles using Whisper
|
|
186
|
+
* Automatically translate to Spanish, French, and German
|
|
187
|
+
* Preserve timing information from the original subtitles
|
|
188
|
+
* Create separate SRT files for each language in `build/`
|
|
189
|
+
|
|
190
|
+
3. **Build your video with all subtitle tracks:**
|
|
191
|
+
|
|
192
|
+
```bash
|
|
193
|
+
mkzforge build
|
|
194
|
+
```
|
|
195
|
+
|
|
196
|
+
The final video will contain all subtitle tracks, properly mapped and labeled.
|
|
197
|
+
|
|
198
|
+
### How Translation Works
|
|
199
|
+
|
|
200
|
+
**Context-Aware Translation**: The entire transcript is translated as one document to preserve
|
|
201
|
+
meaning, intent, and context. This produces much better results than translating line-by-line.
|
|
202
|
+
|
|
203
|
+
**Timing Preservation**: After translation, the system intelligently splits the translated text to
|
|
204
|
+
match the original subtitle timing, distributing words proportionally across subtitle entries.
|
|
205
|
+
|
|
206
|
+
**Automatic Package Management**: Argos Translate language packages are downloaded and installed
|
|
207
|
+
automatically on first use.
|
|
208
|
+
|
|
209
|
+
### Example Output
|
|
210
|
+
|
|
211
|
+
After running `mkzforge refresh` with multi-language support:
|
|
212
|
+
|
|
213
|
+
```plain
|
|
214
|
+
build/
|
|
215
|
+
├── my-video.mkv # Processed video
|
|
216
|
+
├── my-video.txt # Full English transcript
|
|
217
|
+
├── my-video.en.srt # English subtitles (Whisper)
|
|
218
|
+
├── my-video.es.srt # Spanish translation
|
|
219
|
+
├── my-video.fr.srt # French translation
|
|
220
|
+
├── my-video.de.srt # German translation
|
|
221
|
+
└── my-video.mp4 # Final video with all subtitle tracks
|
|
222
|
+
```
|
|
223
|
+
|
|
224
|
+
### Supported Languages
|
|
225
|
+
|
|
226
|
+
Any language pair supported by Argos Translate, including:
|
|
227
|
+
|
|
228
|
+
* Spanish (es), French (fr), German (de), Portuguese (pt)
|
|
229
|
+
* Italian (it), Russian (ru), Chinese (zh), Japanese (ja)
|
|
230
|
+
* Arabic (ar), Hindi (hi), Korean (ko)
|
|
231
|
+
* And many more!
|
|
232
|
+
|
|
233
|
+
For detailed documentation, see [CHANGES.md](CHANGES.md).
|
|
234
|
+
|
|
235
|
+
## @FutureFeature
|
|
236
|
+
|
|
237
|
+
Features I'd like to add to this include:
|
|
238
|
+
|
|
239
|
+
* `mkzforge publish` to publish to your configured social media platforms
|
|
240
|
+
* I want to support YouTube, TikTok, and Mastadon.
|
|
241
|
+
* Right now, only an SFTP endpoint is supported (uses [Fabric](https://www.fabfile.org/)).
|
|
242
|
+
* Stream specifier error clarification: It would be nice to know that a stream is disconnected
|
|
243
|
+
right there in vscode or some editor of sorts. I hope a project like this can make that more
|
|
244
|
+
easily accesible as a potential by simulating the results of the configuration in ffmpeg's
|
|
245
|
+
`filter_complex` parameter.
|