violin 0.1.0__py3-none-any.whl

.claude/skills/video-translator/SKILL.md

@@ -0,0 +1,67 @@
+---
+name: video-translator
+description: Dub a video into another language and generate subtitles using the default Together + Cartesia stack. Trigger when the user wants to translate / dub / voice-over a video file, or generate subtitles for it. Handles `.mp4` / `.mkv` / `.webm`. Installs as the `violin` CLI (and `violin-api` for the FastAPI server) via `uv tool install`. For alternative models (OpenAI / ElevenLabs) or custom configs, point the user to the repo: https://github.com/shang-zhu/violin.
+allowed-tools: Bash, Read
+---
+
+# Violin — operating skill
+
+Always uses the default config (Together for translation, `cartesia/sonic-3` for TTS). If the user asks for OpenAI, ElevenLabs, or custom configs, **stop and point them to the Violin repo** — those flows aren't supported through the global CLI.
+
+## Pre-flight
+
+Run these silently first. Abort if any fails:
+
+```bash
+command -v violin                 # 1. CLI on PATH
+test -f "<input>"                 # 2. Input exists
+printenv TOGETHER_API_KEY         # 3. Key available
+```
+
+If `violin` is missing: tell the user to `uv tool install violin`, then `violin --install-skill` to refresh this skill file. Do not auto-install.
+
+If `TOGETHER_API_KEY` is missing:
+- Inside the Violin repo → populate `.env` (auto-loaded)
+- Elsewhere → `export TOGETHER_API_KEY=...` in `~/.zshrc` / `~/.bashrc`, then `source` it
+
+## Decisions
+
+- **CLI vs API**: single run-and-wait file → CLI (`violin`). Multi-job / HTTP / web UI → API server (`violin-api`); print the command, don't auto-start it.
+- **Style** (`--style`): default `standard`. Kids content → `kids`, formal/lecture → `academic`, casual → `casual`, dramatic → `storyteller`, news → `news`. Run `violin --style list` if unsure.
+- **Voiceover**: keep default (mix dubbed audio over a quiet original). Use `--no-voiceover` only when the user explicitly says "replace audio entirely".
+
+## Run
+
+```bash
+violin <input> <output> --language <Lang> [flags]
+```
+
+## Flags
+
+| Flag | Default | When to set |
+|------|---------|-------------|
+| `--language` / `-l` | *required* | Target language (e.g. `Chinese`, `Spanish`, `Japanese`). |
+| `--voice` / `-v` | auto (native voice picked by `preferences.voice_gender`) | Only when the user names a specific voice from the catalog (e.g. `"warm female narrator"`). Otherwise omit and let the default kick in. |
+| `--source-language` | `auto-detect` | Only if Whisper mis-detects the source language. |
+| `--style` / `-s` | `standard` | See Decisions above. |
+| `--no-subtitles` | off | User says "no SRT" / "video only". |
+| `--no-voiceover` | off | User says "replace original audio entirely". |
+| `--config` / `-c` | `config/default.yaml` | Don't use through this skill — repo-only flow. |
+| `--timings-out` | off | Only when the user wants a per-step timing JSON for debugging / benchmarking. |
+
+## Language coverage
+
+33 target languages total. **16** ship with handpicked native-speaker voices: Chinese, Spanish, English, Hindi, Arabic, Portuguese, Russian, Japanese, Turkish, German, Korean, French, Italian, Polish, Dutch, Swedish. The other **17** fall back to the English voice catalog (multilingual under Cartesia Sonic 3) — quality is decent but the voice isn't a native speaker. Mention this caveat only if the user is translating to a fallback language and asks about voice quality.
+
+## Report back
+
+- Output video path + SRT path (printed by the run).
+- Total cost (printed at end — surface, don't hide).
+- If voiceover was on, mention the `_original.m4a` sidecar.
+
+## Don'ts
+
+- Don't run on multi-GB videos without first quoting the rough cost (audio length × per-provider rates in `pipeline/pricing.py`).
+- Don't fabricate a "subtitles-only" mode — the CLI requires the full pipeline. If the user only wants SRT, run the full pipeline and hand them just the `.srt`, warning them of the cost first.
+- Don't try to switch to OpenAI or ElevenLabs from this skill. Point the user to the repo + `--config config/other_api.yaml` (or their own override).
+- Don't paraphrase the README. For supported languages (33), voice catalog, and full flag docs, point them at `README.md` or `violin --help`.

api/app.py

@@ -0,0 +1,109 @@
+"""FastAPI application factory."""
+
+import asyncio
+import logging
+import os
+import pathlib
+
+from fastapi import FastAPI
+from fastapi.middleware.cors import CORSMiddleware
+from fastapi.responses import FileResponse
+from fastapi.staticfiles import StaticFiles
+
+from .routes import catalog, chat, files, jobs
+
+logger = logging.getLogger(__name__)
+
+_STATIC = pathlib.Path(__file__).parent / "static"
+
+_ALLOWED_ORIGINS = os.environ.get("CORS_ORIGINS", "*").split(",")
+
+app = FastAPI(
+    title="Violin API",
+    description=(
+        "Translate educational videos into 42 languages using Together AI. "
+        "Upload a video, poll for status, then download the dubbed output."
+    ),
+    version="0.1.0",
+    docs_url="/docs",
+    redoc_url="/redoc",
+)
+
+app.add_middleware(
+    CORSMiddleware,
+    allow_origins=_ALLOWED_ORIGINS,
+    allow_methods=["*"],
+    allow_headers=["*"],
+)
+
+app.include_router(jobs.router)
+app.include_router(files.router)
+app.include_router(catalog.router)
+app.include_router(chat.router)
+
+app.mount("/static", StaticFiles(directory=str(_STATIC)), name="static")
+
+
+@app.get("/", include_in_schema=False)
+def root():
+    # Force the SPA shell to revalidate on every load — otherwise users keep
+    # seeing stale UI after upgrading violin-api (esp. on 127.0.0.1 vs localhost,
+    # which Chrome caches as separate origins).
+    return FileResponse(
+        str(_STATIC / "index.html"),
+        headers={"Cache-Control": "no-cache, must-revalidate"},
+    )
+
+
+# Lightweight health probe — accepts both GET and HEAD so external monitors
+# (e.g. UptimeRobot's free tier, which only does HEAD) don't get 405s.
+@app.api_route("/health", methods=["GET", "HEAD"], include_in_schema=False)
+def health():
+    return {"ok": True}
+
+
+@app.get("/app-config", include_in_schema=False)
+def app_config():
+    from .config import (
+        FREE_TRIAL_JOBS,
+        JOB_TTL_HOURS,
+        MAX_DURATION_SECONDS,
+        MAX_FILE_SIZE_MB,
+        URL_UPLOAD,
+    )
+    return {
+        "url_upload": URL_UPLOAD,
+        "max_duration_seconds": MAX_DURATION_SECONDS,
+        "max_file_size_mb": MAX_FILE_SIZE_MB,
+        # Used by the footer to hide privacy/auto-delete warnings on local deployments.
+        "free_trial_jobs": FREE_TRIAL_JOBS,
+        "job_ttl_hours": JOB_TTL_HOURS,
+    }
+
+
+@app.on_event("startup")
+async def _init_stats():
+    """Create the stats table if it doesn't exist. Idempotent — existing rows
+    are preserved across restarts."""
+    from . import stats as _stats
+    _stats.init_db()
+
+
+@app.on_event("startup")
+async def _start_cleanup_loop():
+    from .config import JOB_TTL_HOURS
+    if JOB_TTL_HOURS <= 0:
+        return
+
+    async def _cleanup_loop():
+        from .storage import cleanup_old_jobs
+        while True:
+            await asyncio.sleep(3600)
+            try:
+                deleted = cleanup_old_jobs(JOB_TTL_HOURS)
+                if deleted:
+                    logger.info("Cleaned up %d expired job(s)", deleted)
+            except Exception:
+                logger.exception("Job cleanup failed")
+
+    asyncio.create_task(_cleanup_loop())

api/config.py

@@ -0,0 +1,15 @@
+"""API configuration."""
+
+from pathlib import Path
+
+from pipeline import config as _conf
+
+_api = _conf.get()["api"]
+
+JOBS_DIR = Path(_api["jobs_dir"])
+MAX_WORKERS = _api["max_workers"]
+JOB_TTL_HOURS = _api.get("job_ttl_hours", 24)
+FREE_TRIAL_JOBS = _api.get("free_trial_jobs", 1)
+URL_UPLOAD = _api.get("url_upload", True)
+MAX_DURATION_SECONDS = _api.get("max_duration_seconds", 1800)
+MAX_FILE_SIZE_MB = _api.get("max_file_size_mb", 500)

api/models.py

@@ -0,0 +1,87 @@
+"""Pydantic models for API requests and responses."""
+
+from __future__ import annotations
+
+from enum import Enum
+from typing import Any
+
+from pydantic import BaseModel, Field
+
+
+class JobStatus(str, Enum):
+    queued = "queued"
+    running = "running"
+    done = "done"
+    failed = "failed"
+    cancelled = "cancelled"
+
+
+class ProgressEvent(BaseModel):
+    step: int
+    total: int
+    message: str
+
+
+class JobResponse(BaseModel):
+    id: str
+    status: JobStatus
+    language: str
+    voice: str
+    source_language: str
+    subtitles: bool
+    style: str = "standard"
+    progress: list[ProgressEvent] = Field(default_factory=list)
+    error: str | None = None
+
+
+class CreateJobRequest(BaseModel):
+    """Used internally — the route uses Form() fields directly."""
+    language: str
+    voice: str = ""
+    source_language: str = "auto-detect"
+    subtitles: bool = True
+
+
+class SubtitleSegment(BaseModel):
+    id: int
+    start: float
+    end: float
+    text: str
+    speaker: str = "SPEAKER_00"
+
+
+class ChatMessage(BaseModel):
+    role: str
+    content: str
+
+
+class VoiceMatchRequest(BaseModel):
+    description: str
+    language: str = ""
+    together_api_key: str = ""
+    openai_api_key: str = ""
+
+
+class VoiceCandidate(BaseModel):
+    voice: str
+    explanation: str
+
+
+class VoiceMatchResponse(BaseModel):
+    candidates: list[VoiceCandidate]
+
+
+class VideoChatRequest(BaseModel):
+    question: str
+    current_time: float = Field(ge=0)
+    history: list[ChatMessage] = Field(default_factory=list)
+    language: str = ""
+
+
+class VideoChatResponse(BaseModel):
+    answer: str
+    context_start: float
+    context_end: float
+    subtitle_context: list[SubtitleSegment] = Field(default_factory=list)
+    sampled_timestamps: list[float] = Field(default_factory=list)
+    style: str = "standard"

api/routes/catalog.py

@@ -0,0 +1,190 @@
+"""Catalog endpoints: list supported languages, voices, and styles."""
+
+from __future__ import annotations
+
+import json
+import os
+import re
+
+from dotenv import load_dotenv
+from fastapi import APIRouter, HTTPException
+
+from api.models import VoiceCandidate, VoiceMatchRequest, VoiceMatchResponse
+from pipeline import config as _conf
+from pipeline.languages import all_languages, language_code
+from pipeline.llm_client import get_translation_model, make_translation_client
+from pipeline.styles import list_styles
+from pipeline.tts import all_voices, native_voices_for, voice_descriptions
+import prompts as _prompts
+
+load_dotenv(override=True)
+
+router = APIRouter(tags=["catalog"])
+
+
+@router.get("/languages")
+def list_languages() -> dict[str, str]:
+    """Return a mapping of language name → BCP-47 code for all supported languages."""
+    return all_languages()
+
+
+@router.get("/voices")
+def list_voices() -> dict[str, list[str]]:
+    """Return all known native Cartesia Sonic 3 voices grouped by BCP-47 language code."""
+    return all_voices()
+
+
+@router.get("/voices/{language}")
+def voices_for_language(language: str) -> list[str]:
+    """Return native voices for a specific language name or BCP-47 code."""
+    return native_voices_for(language_code(language))
+
+
+@router.get("/styles")
+def get_styles() -> list[dict]:
+    """Return all available translation style profiles."""
+    return [
+        {
+            "name": s.name,
+            "description": s.description,
+            "tts_speed": s.tts_speed,
+            "tts_emotion": s.tts_emotion,
+        }
+        for s in list_styles()
+    ]
+
+
+def _build_voice_catalog(target_lang: str) -> str:
+    """Format the active provider's voice catalog as a string for the LLM prompt.
+
+    Each voice is rendered as `- <name> — <description>`. For Cartesia the
+    description is the name itself (e.g. 'german conversational woman'); for
+    ElevenLabs it is the official metadata description.
+    """
+    voices = all_voices()
+    descriptions = voice_descriptions()
+    target_code = language_code(target_lang) if target_lang else ""
+    lines: list[str] = []
+
+    def _fmt(name: str) -> str:
+        d = descriptions.get(name, "")
+        return f"  - {name} — {d}" if d and d != name else f"  - {name}"
+
+    if target_code and target_code in voices:
+        lines.append(f"== Voices for target language ({target_code}) ==")
+        for v in voices[target_code]:
+            lines.append(_fmt(v))
+        lines.append("")
+
+    for code, voice_list in sorted(voices.items()):
+        if code == target_code:
+            continue
+        header = "All voices (multilingual)" if code == "multi" else code
+        lines.append(f"== {header} ==")
+        for v in voice_list:
+            lines.append(_fmt(v))
+    return "\n".join(lines)
+
+
+_MAX_VOICE_MATCH_RETRIES = 3
+
+
+def _parse_voice_candidates(raw: str, all_ids: list[str]) -> list[VoiceCandidate]:
+    """Try to extract a list of VoiceCandidates from the raw LLM response."""
+    raw = re.sub(r"<think>.*?</think>", "", raw, flags=re.DOTALL).strip()
+    raw = raw.removeprefix("```json").removeprefix("```").removesuffix("```").strip()
+
+    bracket_match = re.search(r"\[.*\]", raw, flags=re.DOTALL)
+    if bracket_match:
+        raw = bracket_match.group(0)
+
+    items = json.loads(raw)
+    if isinstance(items, dict):
+        items = [items]
+
+    all_lower = {v.lower(): v for v in all_ids}
+
+    def _normalize(name: str) -> str | None:
+        return all_lower.get(name.lower().strip())
+
+    candidates = []
+    seen: set[str] = set()
+    for item in items:
+        if isinstance(item, str):
+            voice, explanation = _normalize(item), ""
+        else:
+            voice = _normalize(item.get("voice", ""))
+            explanation = item.get("explanation", "")
+        if voice and voice.lower() not in seen:
+            seen.add(voice.lower())
+            candidates.append(VoiceCandidate(voice=voice, explanation=explanation))
+        if len(candidates) == 3:
+            break
+    return candidates
+
+
+@router.post("/voice-match", response_model=VoiceMatchResponse)
+def match_voice(payload: VoiceMatchRequest):
+    """Use an LLM to map a natural language voice description to the best voice in the catalog.
+
+    Reuses the translation client + model (``models.translation``) — no separate
+    configuration needed.
+    """
+    cfg = _conf.get()
+    try:
+        client = make_translation_client(
+            cfg,
+            together_key_override=payload.together_api_key.strip() or None,
+            openai_key_override=payload.openai_api_key.strip() or None,
+        )
+    except RuntimeError as exc:
+        raise HTTPException(status_code=500, detail=str(exc))
+
+    catalog = _build_voice_catalog(payload.language)
+    all_ids: list[str] = []
+    for v_list in all_voices().values():
+        all_ids.extend(v_list)
+    all_ids = list(dict.fromkeys(all_ids))
+
+    messages = [
+        {
+            "role": "system",
+            "content": _prompts.load("voice_match", "system", catalog=catalog),
+        },
+        {
+            "role": "user",
+            "content": _prompts.load(
+                "voice_match", "user",
+                language=payload.language or "not specified",
+                description=payload.description,
+            ),
+        },
+    ]
+
+    # voice_match reuses the translation client + model — same LLM, same provider.
+    model = get_translation_model(cfg)
+
+    last_error = ""
+    for attempt in range(_MAX_VOICE_MATCH_RETRIES):
+        try:
+            response = client.chat.completions.create(
+                model=model,
+                messages=messages,
+                temperature=0.1,
+                max_tokens=400,
+            )
+            msg = response.choices[0].message
+            raw = (msg.content or "").strip()
+
+            if not raw:
+                last_error = "LLM returned empty response"
+                continue
+
+            candidates = _parse_voice_candidates(raw, all_ids)
+            if candidates:
+                return VoiceMatchResponse(candidates=candidates)
+            last_error = "parsed JSON but found no valid voice names"
+        except (json.JSONDecodeError, Exception) as exc:
+            last_error = str(exc)
+
+    raise HTTPException(status_code=502, detail=f"Voice matching failed after {_MAX_VOICE_MATCH_RETRIES} attempts ({last_error})")

api/routes/chat.py

@@ -0,0 +1,39 @@
+"""Video chat endpoints for completed jobs."""
+
+from __future__ import annotations
+
+from fastapi import APIRouter, HTTPException
+
+from api.models import SubtitleSegment, VideoChatRequest, VideoChatResponse
+from api.storage import get_job, load_segments
+from api.video_chat import answer_video_question
+
+router = APIRouter(prefix="/jobs", tags=["chat"])
+
+
+@router.get("/{job_id}/segments", response_model=list[SubtitleSegment])
+def get_job_segments(job_id: str):
+    job = get_job(job_id)
+    if job is None:
+        raise HTTPException(status_code=404, detail=f"Job '{job_id}' not found.")
+    if job.status != "done":
+        raise HTTPException(status_code=409, detail=f"Job '{job_id}' is not complete.")
+    return [SubtitleSegment(**item) for item in load_segments(job_id)]
+
+
+@router.post("/{job_id}/chat", response_model=VideoChatResponse)
+def chat_with_video(job_id: str, payload: VideoChatRequest):
+    try:
+        return answer_video_question(
+            job_id=job_id,
+            question=payload.question,
+            current_time=payload.current_time,
+            history=payload.history,
+            language=payload.language,
+        )
+    except FileNotFoundError as exc:
+        raise HTTPException(status_code=404, detail=str(exc)) from exc
+    except RuntimeError as exc:
+        raise HTTPException(status_code=409, detail=str(exc)) from exc
+    except Exception as exc:
+        raise HTTPException(status_code=500, detail=str(exc)) from exc

api/routes/files.py

@@ -0,0 +1,133 @@
+"""File download endpoints for completed jobs."""
+
+from __future__ import annotations
+
+import subprocess
+
+from fastapi import APIRouter, HTTPException, Query
+from fastapi.responses import FileResponse
+
+from api.models import JobStatus
+from api.storage import get_job, original_audio_path, output_srt_path, output_video_path, voiceover_video_path
+from pipeline.ffmpeg_utils import FFMPEG_EXE
+
+router = APIRouter(prefix="/jobs", tags=["files"])
+
+
+def _assert_done(job_id: str) -> None:
+    job = get_job(job_id)
+    if job is None:
+        raise HTTPException(status_code=404, detail=f"Job '{job_id}' not found.")
+    if job.status != JobStatus.done:
+        raise HTTPException(
+            status_code=409,
+            detail=f"Job '{job_id}' is not complete (status: {job.status}).",
+        )
+
+
+@router.get("/{job_id}/video", response_class=FileResponse)
+def download_video(job_id: str):
+    """Download the dubbed output video. Only available when status=done."""
+    _assert_done(job_id)
+    path = output_video_path(job_id)
+    if not path.exists():
+        raise HTTPException(status_code=404, detail="Output video not found.")
+    return FileResponse(
+        path=str(path),
+        media_type="video/mp4",
+        filename=f"{job_id}_dubbed.mp4",
+    )
+
+
+@router.get("/{job_id}/original-audio")
+def get_original_audio(job_id: str):
+    """Serve the original audio track (aligned to the dubbed timeline) for voice-over mixing."""
+    _assert_done(job_id)
+    path = original_audio_path(job_id)
+    if not path.exists():
+        raise HTTPException(
+            status_code=404,
+            detail="Original audio track not available. The job may not have used voice-over mode.",
+        )
+    return FileResponse(
+        path=str(path),
+        media_type="audio/mp4",
+        filename=f"{job_id}_original.m4a",
+    )
+
+
+@router.get("/{job_id}/video-voiceover", response_class=FileResponse)
+def download_voiceover_video(
+    job_id: str,
+    volume: float = Query(0.1, ge=0.0, le=1.0, description="Original audio volume (0.0–1.0)"),
+):
+    """Download the dubbed video with original audio mixed in at the given volume."""
+    _assert_done(job_id)
+
+    dubbed = output_video_path(job_id)
+    orig_audio = original_audio_path(job_id)
+    if not dubbed.exists():
+        raise HTTPException(status_code=404, detail="Output video not found.")
+    if not orig_audio.exists():
+        raise HTTPException(status_code=404, detail="Original audio not available. Job may not have used voice-over mode.")
+
+    out = voiceover_video_path(job_id)
+    if not out.exists() or not _volume_matches(job_id, volume):
+        _mix_voiceover(str(dubbed), str(orig_audio), str(out), volume)
+        _save_volume(job_id, volume)
+
+    return FileResponse(
+        path=str(out),
+        media_type="video/mp4",
+        filename=f"{job_id}_voiceover.mp4",
+    )
+
+
+def _mix_voiceover(video_path: str, audio_path: str, output_path: str, volume: float) -> None:
+    """Mix original audio into the dubbed video using ffmpeg."""
+    subprocess.run([
+        FFMPEG_EXE,
+        "-i", video_path,
+        "-i", audio_path,
+        "-filter_complex",
+        f"[0:a]volume=1.0[dub];[1:a]volume={volume}[orig];[dub][orig]amix=inputs=2:duration=first[out]",
+        "-map", "0:v",
+        "-map", "[out]",
+        "-c:v", "copy",
+        "-c:a", "aac",
+        "-movflags", "+faststart",
+        "-y", output_path,
+    ], check=True, capture_output=True)
+
+
+def _volume_matches(job_id: str, volume: float) -> bool:
+    """Check if the cached voiceover was mixed at the same volume."""
+    vol_file = voiceover_video_path(job_id).with_suffix(".vol")
+    if not vol_file.exists():
+        return False
+    try:
+        return abs(float(vol_file.read_text().strip()) - volume) < 0.001
+    except (ValueError, OSError):
+        return False
+
+
+def _save_volume(job_id: str, volume: float) -> None:
+    vol_file = voiceover_video_path(job_id).with_suffix(".vol")
+    vol_file.write_text(str(volume))
+
+
+@router.get("/{job_id}/srt", response_class=FileResponse)
+def download_srt(job_id: str):
+    """Download the SRT subtitle file. Only available when status=done and subtitles=true."""
+    _assert_done(job_id)
+    path = output_srt_path(job_id)
+    if not path.exists():
+        raise HTTPException(
+            status_code=404,
+            detail="SRT file not found. The job may have been created with subtitles=false.",
+        )
+    return FileResponse(
+        path=str(path),
+        media_type="text/plain; charset=utf-8",
+        filename=f"{job_id}.srt",
+    )