violin 0.1.0a1__tar.gz

violin-0.1.0a1/.claude/skills/video-translator/SKILL.md

@@ -0,0 +1,67 @@
+---
+name: video-translator
+description: Dub a video into another language and generate subtitles using the default Together + Cartesia stack. Trigger when the user wants to translate / dub / voice-over a video file, or generate subtitles for it. Handles `.mp4` / `.mkv` / `.webm`. Installs as the `violin` CLI (and `violin-api` for the FastAPI server) via `uv tool install`. For premium models (OpenAI / ElevenLabs) or custom configs, point the user to the repo.
+allowed-tools: Bash, Read
+---
+
+# Violin — operating skill
+
+Always uses the default config (Together for translation, `cartesia/sonic-3` for TTS). If the user asks for OpenAI, ElevenLabs, or custom configs, **stop and point them to the Violin repo** — those flows aren't supported through the global CLI.
+
+## Pre-flight
+
+Run these silently first. Abort if any fails:
+
+```bash
+command -v violin                 # 1. CLI on PATH
+test -f "<input>"                 # 2. Input exists
+printenv TOGETHER_API_KEY         # 3. Key available
+```
+
+If `violin` is missing: tell the user to `uv tool install .` from a Violin checkout. Do not auto-install.
+
+If `TOGETHER_API_KEY` is missing:
+- Inside the Violin repo → populate `.env` (auto-loaded)
+- Elsewhere → `export TOGETHER_API_KEY=...` in `~/.zshrc` / `~/.bashrc`, then `source` it
+
+## Decisions
+
+- **CLI vs API**: single run-and-wait file → CLI (`violin`). Multi-job / HTTP / web UI → API server (`violin-api`); print the command, don't auto-start it.
+- **Style** (`--style`): default `standard`. Kids content → `kids`, formal/lecture → `academic`, casual → `casual`, dramatic → `storyteller`, news → `news`. Run `violin --style list` if unsure.
+- **Voiceover**: keep default (mix dubbed audio over a quiet original). Use `--no-voiceover` only when the user explicitly says "replace audio entirely".
+
+## Run
+
+```bash
+violin <input> <output> --language <Lang> [flags]
+```
+
+## Flags
+
+| Flag | Default | When to set |
+|------|---------|-------------|
+| `--language` / `-l` | *required* | Target language (e.g. `Chinese`, `Spanish`, `Japanese`). |
+| `--voice` / `-v` | auto (native voice picked by `preferences.voice_gender`) | Only when the user names a specific voice from the catalog (e.g. `"warm female narrator"`). Otherwise omit and let the default kick in. |
+| `--source-language` | `auto-detect` | Only if Whisper mis-detects the source language. |
+| `--style` / `-s` | `standard` | See Decisions above. |
+| `--no-subtitles` | off | User says "no SRT" / "video only". |
+| `--no-voiceover` | off | User says "replace original audio entirely". |
+| `--config` / `-c` | `config/default.yaml` | Don't use through this skill — repo-only flow. |
+| `--timings-out` | off | Only when the user wants a per-step timing JSON for debugging / benchmarking. |
+
+## Language coverage
+
+33 target languages total. **16** ship with handpicked native-speaker voices: Chinese, Spanish, English, Hindi, Arabic, Portuguese, Russian, Japanese, Turkish, German, Korean, French, Italian, Polish, Dutch, Swedish. The other **17** fall back to the English voice catalog (multilingual under Cartesia Sonic 3) — quality is decent but the voice isn't a native speaker. Mention this caveat only if the user is translating to a fallback language and asks about voice quality.
+
+## Report back
+
+- Output video path + SRT path (printed by the run).
+- Total cost (printed at end — surface, don't hide).
+- If voiceover was on, mention the `_original.m4a` sidecar.
+
+## Don'ts
+
+- Don't run on multi-GB videos without first quoting the rough cost (audio length × per-provider rates in `pipeline/pricing.py`).
+- Don't fabricate a "subtitles-only" mode — the CLI requires the full pipeline. If the user only wants SRT, run the full pipeline and hand them just the `.srt`, warning them of the cost first.
+- Don't try to switch to OpenAI or ElevenLabs from this skill. Point the user to the repo + `--config config/other_api.yaml` (or their own override).
+- Don't paraphrase the README. For supported languages (33), voice catalog, and full flag docs, point them at `README.md` or `violin --help`.

violin-0.1.0a1/.dockerignore

@@ -0,0 +1,17 @@
+.git
+.gitignore
+.env
+.env.*
+!.env.example
+jobs/
+__pycache__
+*.pyc
+*.pyo
+.venv
+.mypy_cache
+.ruff_cache
+*.egg-info
+dist/
+build/
+*.md
+!README.md

violin-0.1.0a1/.env.example

@@ -0,0 +1,21 @@
+# ── API keys ─────────────────────────────────────────────────
+# Only fill in the keys for the providers you actually use — see
+# config/default.yaml `models:` section. The default Together stack
+# (Whisper + LLM + Cartesia TTS) needs only TOGETHER_API_KEY.
+
+# Together AI — https://api.together.ai
+TOGETHER_API_KEY=
+
+# OpenAI — needed when any stage uses `provider: openai`
+# (whisper-1, GPT models, tts-1, vision chat). https://platform.openai.com
+OPENAI_API_KEY=
+
+# ElevenLabs — needed when TTS uses `provider: elevenlabs`
+# https://elevenlabs.io/app/settings/api-keys
+ELEVENLABS_API_KEY=
+
+
+# ── Optional ─────────────────────────────────────────────────
+
+# CORS allow-list for the web app (default: * — any origin)
+# CORS_ORIGINS=https://yourdomain.com

violin-0.1.0a1/.gitignore

@@ -0,0 +1,18 @@
+# Python-generated files
+__pycache__/
+*.py[oc]
+build/
+dist/
+wheels/
+*.egg-info
+
+# Virtual environments
+.venv
+.env
+examples/
+
+#logs
+*.log
+jobs/
+
+.DS_Store

violin-0.1.0a1/.python-version

@@ -0,0 +1 @@
+3.13

violin-0.1.0a1/Caddyfile

@@ -0,0 +1,12 @@
+# Replace with your actual domain name.
+# Caddy will automatically provision a Let's Encrypt TLS certificate
+# once the domain's DNS A record points to this server's public IP.
+#
+# For local testing without a domain, replace with:
+#   :80 {
+#       reverse_proxy violin:8000
+#   }
+
+violin-ai.com {
+    reverse_proxy violin:8000
+}

violin-0.1.0a1/Dockerfile

@@ -0,0 +1,19 @@
+FROM python:3.13-slim AS base
+
+RUN apt-get update && \
+    apt-get install -y --no-install-recommends libsndfile1 ffmpeg && \
+    rm -rf /var/lib/apt/lists/*
+
+COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv
+
+WORKDIR /app
+
+COPY pyproject.toml uv.lock ./
+RUN uv sync --frozen --no-dev --no-install-project
+
+COPY . .
+RUN uv sync --frozen --no-dev
+
+EXPOSE 8000
+
+CMD ["uv", "run", "run_api.py", "--host", "0.0.0.0", "--port", "8000", "--config", "config/prod.yaml"]

violin-0.1.0a1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Violin contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

violin-0.1.0a1/PKG-INFO

@@ -0,0 +1,298 @@
+Metadata-Version: 2.4
+Name: violin
+Version: 0.1.0a1
+Summary: Open-source video dubbing — translate any video into 33 languages with native-sounding voice-over and synced subtitles.
+Project-URL: Repository, https://github.com/shang-zhu/Violin
+Project-URL: Issues, https://github.com/shang-zhu/Violin/issues
+Author: Shang Zhu, Qinghong Lin
+License: MIT
+License-File: LICENSE
+Keywords: dubbing,elevenlabs,subtitles,together-ai,translation,tts,video,whisper
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: End Users/Desktop
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Multimedia :: Sound/Audio :: Speech
+Classifier: Topic :: Multimedia :: Video
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.13
+Requires-Dist: aiofiles>=25.1.0
+Requires-Dist: elevenlabs>=1.0.0
+Requires-Dist: fastapi>=0.135.2
+Requires-Dist: ffmpeg-python>=0.2.0
+Requires-Dist: httpx>=0.28.1
+Requires-Dist: imageio-ffmpeg>=0.6.0
+Requires-Dist: openai>=1.0.0
+Requires-Dist: python-dotenv>=1.2.2
+Requires-Dist: python-multipart>=0.0.22
+Requires-Dist: pyyaml>=6.0.3
+Requires-Dist: soundfile>=0.13.1
+Requires-Dist: together>=2.5.0
+Requires-Dist: uvicorn>=0.42.0
+Requires-Dist: yt-dlp>=2025.1.0
+Description-Content-Type: text/markdown
+
+# 🎻 Violin
+
+**Open-source video dubbing — translate any video into 33 languages with natural-sounding voice-over and synced subtitles.**
+
+[🌐 Live demo](https://violin-ai.com) · [📜 MIT License](LICENSE)
+
+<!-- ![demo](assets/outcome.png) -->
+
+Upload a video. Violin transcribes the speech, translates it, synthesizes a native-sounding voice-over in the target language, and remuxes it back into the video — fully aligned, with optional SRT subtitles.
+
+Available as a **CLI**, a **FastAPI web app**, and a **Claude Code skill**.
+
+---
+
+## ✨ Features
+
+- **33 target languages** with handpicked native-speaker voices for the 16 most-used ones (Cartesia Sonic 3 + ElevenLabs)
+- **In-video Q&A** — ask questions about any moment in the dubbed video; answers use nearby subtitles plus sampled frames
+- **Natural-language voice picker** — describe the voice you want, an LLM picks from the catalog
+- **6 style profiles** *(experimental)* — standard / kids / academic / casual / storyteller / news
+- **Pluggable stack** — Together / OpenAI / ElevenLabs interchangeable for every stage, one YAML
+
+---
+
+## 🚀 Quick start
+
+### Try it without installing anything
+
+The live demo runs at **<https://violin-ai.com>** — drop a short clip in, get a dubbed video out in a few minutes.
+
+### Run locally
+
+Requires **Python 3.13+** and **ffmpeg** on PATH.
+
+```bash
+git clone https://github.com/shang-zhu/violin.git
+cd violin
+uv sync
+cp .env.example .env          # then fill in TOGETHER_API_KEY (get one at https://api.together.ai)
+```
+
+Three ways to use it:
+
+**1. CLI** — translate one file:
+
+```bash
+uv run main.py assets/demo_en.mp4.mp4 assets/demo_en_zh.mp4 --language Chinese
+```
+
+**2. Web app** — full REST API + browser UI:
+
+```bash
+uv run run_api.py
+# → http://127.0.0.1:8000           (browser UI)
+# → http://127.0.0.1:8000/docs      (interactive API docs)
+```
+
+**3. Claude Code skill** — invoke from any Claude Code session:
+
+```bash
+cp -r .claude/skills/video-translator ~/.claude/skills/
+claude
+> please use the violin skill to translate assets/demo_en.mp4 into Chinese
+```
+
+---
+
+## 🎬 How Violin works
+
+```
+Video
+  │
+  ├─ ffmpeg ─────────────────────► Extract audio (16 kHz WAV)
+  │
+  ├─ Whisper Large v3 ────────────► Word-level timestamps → sentence segments
+  │
+  ├─ LLM (DeepSeek V4 Pro by default) ──► Translate each segment, respecting style profile
+  │
+  ├─ TTS (Cartesia Sonic 3 by default) ─► Synthesize dubbed audio per segment
+  │
+  └─ ffmpeg ─────────────────────► Speed-align video to dubbed audio,
+                                    concat with freeze-frame fallback,
+                                    single-pass AAC encode the audio track,
+                                    write output mp4 + optional SRT
+```
+
+Key engineering decisions worth a look if you're forking:
+
+- **`pipeline/transcriber.py`** — uses Whisper's word-level timestamps to split into precise sentence boundaries. Has a hallucination filter that re-uses Whisper's own `no_speech_prob` segment metadata (no hand-tuned heuristics).
+- **`pipeline/merger.py`** — concatenates speed-adjusted video chunks, but builds the audio track *once* from concatenated PCM and encodes AAC at the end. This is the difference between "subtitles drift 1–2 s by the end of an 8 min video" and "perfectly synced throughout."
+- **`pipeline/tts_*.py`** — Cartesia + ElevenLabs backends share an interface. ElevenLabs side ships 21 premade voices (multilingual via `eleven_v3`) plus 15 hand-picked native-speaker voices from the Voice Library.
+
+---
+
+## ⚙️ Configuration
+
+All defaults live in `config/default.yaml`. Override with `--config my.yaml` (only the keys you want to change need to appear in the override file — values deep-merge).
+
+### Switch providers
+
+```yaml
+# config/default.yaml — pick the stack you want
+models:
+  transcription:
+    provider: together                  # together | openai
+    model: openai/whisper-large-v3      # together → openai/whisper-large-v3 | openai → whisper-1
+  translation:
+    provider: together                  # together | openai
+    model: deepseek-ai/DeepSeek-V4-Pro  # together → deepseek-ai/DeepSeek-V4-Pro | openai → gpt-5.5
+  tts:
+    provider: together                  # together | elevenlabs | openai
+    model: cartesia/sonic-3             # together → cartesia/sonic-3 | elevenlabs → eleven_v3 | openai → tts-1-hd
+```
+
+### Production overrides
+
+A starter `config/prod.yaml` is included for public deployments. It adds upload limits, serializes jobs, and caps ffmpeg concurrency. The included `Dockerfile` + `docker-compose.yml` + `Caddyfile` are how the live demo is hosted — `docker compose up -d --build` after filling `.env` is enough to put a copy of Violin behind auto-HTTPS on any Docker host.
+
+### Environment variables
+
+| Variable | When required | Description |
+|----------|---------------|-------------|
+| `TOGETHER_API_KEY` | **Recommended** — covers every stage with the default config | Together AI API key |
+| `OPENAI_API_KEY` | Any stage uses `provider: openai` | Covers `whisper-1`, GPT models, and `tts-1` |
+| `ELEVENLABS_API_KEY` | TTS uses `provider: elevenlabs` | ElevenLabs API key |
+| `CORS_ORIGINS` | Optional | Comma-separated allowed origins (default: `*`) |
+
+> You only need keys for the providers you actually pick. Pure-OpenAI deployments (all stages on `openai`) work too — `OPENAI_API_KEY` alone is enough. Same idea for ElevenLabs.
+
+---
+
+## 🎭 Style profiles
+
+Six built-in profiles tune both the translation LLM prompt and the TTS delivery. Use `--style <name>` on the CLI or pass `style` in API requests.
+
+| Style | Tone | TTS speed | Emotion |
+|-------|------|-----------|---------|
+| `standard` | Faithful translation, natural voice | 1.0× | — |
+| `kids` | Rewritten for a 7-year-old, plain language | 1.0× | excited |
+| `academic` | Formal register, preserves jargon and honorifics | 0.95× | calm |
+| `casual` | Spoken slang, contractions, friendly | 1.1× | content |
+| `storyteller` | Vivid, dramatic narration | 0.9× | enthusiastic |
+| `news` | Concise, declarative, broadcast-style | 1.0× | neutral |
+
+Add your own by editing `prompts/styles.yaml`.
+
+See all available styles: `uv run main.py --style list`.
+
+---
+
+## 💻 CLI usage
+
+```bash
+# Basic
+uv run main.py lecture.mp4 lecture_es.mp4 --language Spanish
+
+# Pick a style
+uv run main.py talk.mp4 talk_zh.mp4 --language Chinese --style kids
+
+# Pick a specific voice
+uv run main.py lecture.mp4 lecture_fr.mp4 --language French --voice "french narrator man"
+
+# Skip SRT
+uv run main.py lecture.mp4 lecture_ja.mp4 --language Japanese --no-subtitles
+
+# Full replacement (no original audio underneath)
+uv run main.py lecture.mp4 lecture_ko.mp4 --language Korean --no-voiceover
+
+# Custom config (e.g. switch to OpenAI/ElevenLabs)
+uv run main.py lecture.mp4 lecture_it.mp4 --language Italian --config config/other_api.yaml
+```
+
+### CLI flags
+
+| Flag | Default | Description |
+|------|---------|-------------|
+| `--language` / `-l` | *(required)* | Target language name (e.g. `Spanish`, `Japanese`) |
+| `--voice` / `-v` | auto | TTS voice. Defaults to the primary native voice for the target language |
+| `--source-language` | `auto-detect` | Source language hint for translation |
+| `--no-subtitles` | off | Skip SRT generation |
+| `--voiceover` / `--no-voiceover` | voiceover on | Keep original audio underneath the dub, or full replacement |
+| `--style` / `-s` | `standard` | Style profile name. Use `--style list` to see all |
+| `--config` / `-c` | `config/default.yaml` | Path to a YAML override file |
+| `--timings-out` | off | Write per-step wall-clock timings + cost as JSON |
+
+---
+
+## 🛰️ Web app & REST API
+
+```bash
+uv run run_api.py                              # default dev mode
+uv run run_api.py --host 0.0.0.0 --port 8080   # bind everywhere
+uv run run_api.py --config config/prod.yaml    # production overrides
+```
+
+Core flow: `POST /jobs` to start, `GET /jobs/{id}` to poll, `GET /jobs/{id}/video` and `/srt` to download, `POST /jobs/{id}/chat` for in-video Q&A. Full list with request/response schemas at **`/docs`**.
+
+### Example
+
+```bash
+# Submit
+JOB=$(curl -s -X POST http://localhost:8000/jobs \
+  -F "file=@lecture.mp4" \
+  -F "language=Spanish" \
+  -F "style=academic" | jq -r .id)
+
+# Poll
+curl -s http://localhost:8000/jobs/$JOB | jq '{status, progress}'
+
+# Download
+curl -OJ http://localhost:8000/jobs/$JOB/video
+curl -OJ http://localhost:8000/jobs/$JOB/srt
+```
+
+Job data lives under `jobs/{id}/`. Set `api.job_ttl_hours` to auto-delete jobs older than N hours (default `0` = disabled; `config/prod.yaml` uses 24h for the public demo).
+
+---
+
+## 🌍 Supported languages
+
+Violin supports **33 target languages**. The 16 below ship with handpicked native-speaker voices for each provider; the rest fall back to the English voice catalog (which is multilingual under both Cartesia Sonic 3 and ElevenLabs `eleven_v3`).
+
+Ordered by native-speaker population.
+
+| Language | Cartesia native voice (M / F) | ElevenLabs native voice (M / F) |
+|----------|-------------------------------|---------------------------------|
+| Chinese | chinese commercial man / chinese female conversational | Lin / Lingyue |
+| Spanish | spanish narrator man / spanish narrator lady | Carlos / Valeria |
+| English | tutorial man / helpful woman | Adam / Sarah |
+| Hindi | hindi narrator man / hindi narrator woman | Yatin / Madhusmita |
+| Arabic | middle eastern woman | Faris / Haneen |
+| Portuguese | friendly brazilian man / pleasant brazilian lady | Medeiros / Luna |
+| Russian | russian narrator man 1 / russian narrator woman | Ivo / Xenia |
+| Japanese | japanese male conversational / japanese woman conversational | Shohei / Maiko |
+| Turkish | turkish narrator man / turkish calm man | Sinan / Aura |
+| German | german reporter man / german conversational woman | Daniel / Sina |
+| Korean | korean narrator man / korean calm woman | Joon-ho / Soo |
+| French | french narrator man / french narrator lady | Lior / Virginie |
+| Italian | italian narrator man / italian narrator woman | Raffaele / Chiara |
+| Polish | polish confident man / polish narrator woman | Gregor / Jola |
+| Dutch | dutch confident man / dutch man | Ronald / Jolanda |
+| Swedish | swedish narrator man / swedish calm lady | Andreas / Louise |
+
+The 17 fallback languages (using the English voice catalog), also ordered by native speakers: Vietnamese, Tamil, Indonesian, Malay, Ukrainian, Romanian, Thai, Greek, Hungarian, Catalan, Czech, Bulgarian, Danish, Slovak, Croatian, Finnish, Norwegian.
+
+---
+
+## 🤝 Contributing
+
+PRs welcome. Got questions or hit a bug? Email **<heyviolinai@gmail.com>** or open an issue.
+
+---
+
+## 📜 License
+
+[MIT](LICENSE) — use it freely, including commercially.
+
+---
+
+## 🙏 Acknowledgements
+
+Built on top of [Together AI](https://together.ai), [Whisper](https://github.com/openai/whisper), [Cartesia Sonic 3](https://cartesia.ai), [ElevenLabs](https://elevenlabs.io), [FastAPI](https://fastapi.tiangolo.com/), and [ffmpeg](https://ffmpeg.org).