vargai 0.3.0

package/ffmpeg/CLAUDE.md

@@ -0,0 +1,68 @@
+# ffmpeg video mixing lessons
+
+## problem: audio/video desync when mixing clips
+
+### what went wrong
+
+using `-ss X -t Y` to pre-trim input, then applying relative trim filters caused timing drift:
+
+```bash
+# BAD: relative timestamps after pre-trim
+ffmpeg -ss 64 -t 36 -i original.mp4 ...
+  -filter_complex "
+    [0:v]split=5[orig1][orig2]...;
+    [orig1]trim=0:4,setpts=PTS-STARTPTS[o1];
+    [orig2]trim=4:11,setpts=PTS-STARTPTS[o2];
+    ..."
+```
+
+this produced wrong duration (38s instead of 36s) with audio desync.
+
+### solution: use absolute timestamps from full input
+
+trim directly from full original using absolute timestamps:
+
+```bash
+# GOOD: absolute timestamps from full file
+ffmpeg -i original.mp4 -i scene1.mp4 -i scene2.mp4 ...
+  -filter_complex "
+    [0:v]trim=64:68,setpts=PTS-STARTPTS[o1];
+    [1:v]scale=1280:720,trim=4:6,setpts=PTS-STARTPTS[s1];
+    [0:v]trim=70:75,setpts=PTS-STARTPTS[o2];
+    ...
+    [o1][s1][o2]...concat=n=N:v=1:a=0[outv];
+    [0:a]atrim=64:100,asetpts=PTS-STARTPTS[outa]
+  "
+  -map "[outv]" -map "[outa]"
+```
+
+### key points
+
+1. **absolute timestamps**: trim from full input file, not pre-trimmed
+2. **separate audio handling**: use `atrim` on audio stream independently
+3. **setpts reset**: always use `setpts=PTS-STARTPTS` after trim to reset timestamps
+4. **scale before trim**: when mixing different resolutions, scale first then trim
+5. **video duration = audio duration**: ensure total video segments match audio segment length
+
+### example: inserting clips into original
+
+to insert generated clips at specific timestamps while keeping continuous audio:
+
+```bash
+ffmpeg -y \
+  -i original.mp4 \
+  -i generated-scene.mp4 \
+  -filter_complex "
+    [0:v]trim=START1:END1,setpts=PTS-STARTPTS[o1];
+    [1:v]scale=1280:720,trim=0:DURATION,setpts=PTS-STARTPTS[s1];
+    [0:v]trim=START2:END2,setpts=PTS-STARTPTS[o2];
+    [o1][s1][o2]concat=n=3:v=1:a=0[outv];
+    [0:a]atrim=AUDIO_START:AUDIO_END,asetpts=PTS-STARTPTS[outa]
+  " \
+  -map "[outv]" -map "[outa]" \
+  -c:v libx264 -preset fast -crf 18 \
+  -c:a aac -b:a 192k \
+  output.mp4
+```
+
+timestamps must add up: `(END1-START1) + DURATION + (END2-START2) = AUDIO_END - AUDIO_START`

package/package.json

@@ -0,0 +1,69 @@
+{
+  "name": "vargai",
+  "module": "src/index.ts",
+  "type": "module",
+  "bin": {
+    "varg": "./src/cli/index.ts"
+  },
+  "scripts": {
+    "check": "biome check . && tsc --noEmit",
+    "lint": "biome check .",
+    "format": "biome format --write .",
+    "type-check": "tsc --noEmit",
+    "prepare": "husky install",
+    "size": "size-limit"
+  },
+  "lint-staged": {
+    "*.{js,ts,tsx}": [
+      "biome check --write --no-errors-on-unmatched"
+    ],
+    "*.json": [
+      "biome format --write"
+    ]
+  },
+  "devDependencies": {
+    "@biomejs/biome": "^2.3.7",
+    "@commitlint/cli": "^20.1.0",
+    "@commitlint/config-conventional": "^20.0.0",
+    "@size-limit/preset-small-lib": "^11.2.0",
+    "@types/bun": "latest",
+    "@types/react": "^19.2.7",
+    "husky": "^9.1.7",
+    "lint-staged": "^16.2.7"
+  },
+  "peerDependencies": {
+    "typescript": "^5"
+  },
+  "dependencies": {
+    "@ai-sdk/fal": "^1.0.23",
+    "@ai-sdk/openai": "^3.0.9",
+    "@ai-sdk/provider": "^3.0.2",
+    "@ai-sdk/replicate": "^2.0.5",
+    "@aws-sdk/client-s3": "^3.937.0",
+    "@aws-sdk/s3-request-presigner": "^3.937.0",
+    "@elevenlabs/elevenlabs-js": "^2.28.0",
+    "@fal-ai/client": "^1.7.2",
+    "@higgsfield/client": "^0.1.2",
+    "@inkjs/ui": "^2.0.0",
+    "@remotion/cli": "^4.0.377",
+    "@types/fluent-ffmpeg": "^2.1.28",
+    "ai": "^6.0.26",
+    "apify-client": "^2.20.0",
+    "citty": "^0.1.6",
+    "fluent-ffmpeg": "^2.1.3",
+    "groq-sdk": "^0.36.0",
+    "ink": "^6.5.1",
+    "react": "^19.2.0",
+    "react-dom": "^19.2.0",
+    "remotion": "^4.0.377",
+    "replicate": "^1.4.0",
+    "zod": "^4.2.1"
+  },
+  "version": "0.3.0",
+  "exports": {
+    ".": "./src/index.ts",
+    "./core": "./src/core/index.ts",
+    "./providers": "./src/providers/index.ts",
+    "./definitions": "./src/definitions/index.ts"
+  }
+}

package/pipeline/cookbooks/SKILL.md

@@ -0,0 +1,285 @@
+---
+name: talking-character-pipeline
+description: complete workflow to create talking character videos with lipsync and captions. use when creating ai character videos, talking avatars, narrated content, or social media character content with voiceover.
+allowed-tools: Read, Bash
+---
+
+# talking character pipeline
+
+create professional talking character videos from scratch using the complete varg.ai sdk workflow.
+
+## overview
+
+this pipeline combines multiple services to create a fully produced talking character video:
+1. character headshot generation
+2. voiceover synthesis
+3. character animation
+4. lipsync
+5. auto-generated captions
+6. social media optimization
+
+**total time**: ~4-5 minutes per video
+
+## step-by-step workflow
+
+### 1. create character headshot
+```bash
+bun run service/image.ts soul "professional headshot of a friendly person, studio lighting" true
+```
+
+**output**: character image url + s3 url  
+**time**: ~30 seconds
+
+**tip**: be specific about character appearance, lighting, and style for best results
+
+### 2. generate voiceover
+```bash
+bun run service/voice.ts elevenlabs "hello world, this is my character speaking" rachel true
+```
+
+**output**: `media/voice-{timestamp}.mp3` + s3 url  
+**time**: ~10 seconds
+
+**tip**: choose voice that matches character (rachel/bella for female, josh/antoni for male)
+
+### 3. animate character
+```bash
+bun run service/video.ts from_image "person talking naturally, professional demeanor" <headshot_url> 5 true
+```
+
+**output**: animated video url + s3 url  
+**time**: ~2-3 minutes
+
+**tip**: use subtle motion prompts like "person talking naturally" or "slight head movement"
+
+### 4. add lipsync
+```bash
+bun run service/sync.ts wav2lip <video_url> <audio_url>
+```
+
+**output**: lipsynced video url  
+**time**: ~30 seconds
+
+**tip**: wav2lip works best with close-up character shots and clear audio
+
+### 5. add captions
+```bash
+bun run service/captions.ts <video_path> captioned.mp4 --provider fireworks
+```
+
+**output**: `captioned.mp4` with subtitles  
+**time**: ~15 seconds (includes transcription)
+
+**tip**: fireworks provider gives word-level timing for professional captions
+
+### 6. prepare for social media
+```bash
+bun run service/edit.ts social captioned.mp4 final-tiktok.mp4 tiktok
+```
+
+**output**: `final-tiktok.mp4` optimized for platform  
+**time**: ~5 seconds
+
+**platforms**: tiktok, instagram, youtube-shorts, youtube, twitter
+
+## complete example
+
+```bash
+# step 1: generate character
+bun run service/image.ts soul \
+  "professional business woman, friendly smile, studio lighting" \
+  true
+
+# step 2: create voiceover
+bun run service/voice.ts elevenlabs \
+  "welcome to our company. we're excited to show you our new product" \
+  rachel \
+  true
+
+# step 3: animate character
+bun run service/video.ts from_image \
+  "person talking professionally" \
+  https://your-s3-url/character.jpg \
+  5 \
+  true
+
+# step 4: sync lips
+bun run service/sync.ts wav2lip \
+  https://your-s3-url/animated.mp4 \
+  https://your-s3-url/voice.mp3
+
+# step 5: add captions
+bun run service/captions.ts \
+  synced-video.mp4 \
+  captioned.mp4 \
+  --provider fireworks \
+  --font "Arial Black" \
+  --size 32
+
+# step 6: optimize for tiktok
+bun run service/edit.ts social \
+  captioned.mp4 \
+  final-tiktok.mp4 \
+  tiktok
+```
+
+## programmatic workflow
+
+```typescript
+import { generateWithSoul } from "./service/image"
+import { generateVoice } from "./service/voice"
+import { generateVideoFromImage } from "./service/video"
+import { lipsyncWav2Lip } from "./service/sync"
+import { addCaptions } from "./service/captions"
+import { prepareForSocial } from "./service/edit"
+
+// 1. character
+const character = await generateWithSoul(
+  "friendly business person, professional",
+  { upload: true }
+)
+
+// 2. voice
+const voice = await generateVoice({
+  text: "hello, welcome to our video",
+  voice: "rachel",
+  upload: true,
+  outputPath: "media/voice.mp3"
+})
+
+// 3. animate
+const video = await generateVideoFromImage(
+  "person talking naturally",
+  character.uploaded!,
+  { duration: 5, upload: true }
+)
+
+// 4. lipsync
+const synced = await lipsyncWav2Lip({
+  videoUrl: video.uploaded!,
+  audioUrl: voice.uploadUrl!
+})
+
+// 5. captions
+const captioned = await addCaptions({
+  videoPath: synced,
+  output: "captioned.mp4",
+  provider: "fireworks"
+})
+
+// 6. social media
+const final = await prepareForSocial({
+  input: captioned,
+  output: "final.mp4",
+  platform: "tiktok"
+})
+```
+
+## use cases
+
+### marketing content
+- product announcements
+- brand messaging
+- explainer videos
+- social media ads
+
+### educational content
+- course introductions
+- tutorial narration
+- lesson summaries
+- educational social media
+
+### social media
+- tiktok character content
+- instagram reels with narration
+- youtube shorts
+- twitter video posts
+
+## tips for best results
+
+**character creation:**
+- be specific about appearance, expression, lighting
+- "professional", "friendly", "casual" work well
+- mention "studio lighting" for clean backgrounds
+
+**voiceover:**
+- write natural, conversational scripts
+- add punctuation for natural pauses
+- keep sentences short and clear
+- match voice gender to character
+
+**animation:**
+- use subtle motion prompts
+- 5 seconds is perfect for character talking shots
+- avoid complex camera movements
+
+**lipsync:**
+- wav2lip works best with frontal face views
+- ensure audio is clear and well-paced
+- close-up shots give better results
+
+**captions:**
+- use fireworks for word-level timing
+- larger font sizes (28-32) work better on mobile
+- white text with black outline is most readable
+
+**social media:**
+- vertical (9:16) for tiktok/instagram/shorts
+- landscape (16:9) for youtube/twitter
+- keep total video under 60 seconds for best engagement
+
+## estimated costs
+
+per video (approximate):
+- character image: $0.05 (higgsfield soul)
+- voiceover: $0.10 (elevenlabs)
+- animation: $0.20 (fal image-to-video)
+- lipsync: $0.10 (replicate wav2lip)
+- transcription: $0.02 (fireworks)
+
+**total**: ~$0.47 per video
+
+## troubleshooting
+
+**character doesn't look consistent:**
+- use higgsfield soul instead of fal for characters
+- save character image and reuse for consistency
+
+**lipsync doesn't match well:**
+- ensure video shows face clearly
+- use close-up shots
+- check audio quality and clarity
+
+**animation looks unnatural:**
+- simplify motion prompt
+- use "person talking naturally" or "slight movement"
+- avoid dramatic camera movements
+
+**captions are off-sync:**
+- use fireworks provider for better timing
+- check audio quality
+- verify video fps is standard (24/30fps)
+
+## required environment variables
+
+```bash
+HIGGSFIELD_API_KEY=hf_xxx
+HIGGSFIELD_SECRET=secret_xxx
+ELEVENLABS_API_KEY=el_xxx
+FAL_API_KEY=fal_xxx
+REPLICATE_API_TOKEN=r8_xxx
+FIREWORKS_API_KEY=fw_xxx
+CLOUDFLARE_R2_API_URL=https://xxx.r2.cloudflarestorage.com
+CLOUDFLARE_ACCESS_KEY_ID=xxx
+CLOUDFLARE_ACCESS_SECRET=xxx
+CLOUDFLARE_R2_BUCKET=m
+```
+
+## next steps
+
+after creating your talking character video:
+- upload to social platforms
+- analyze performance metrics
+- iterate on character design and scripts
+- create series with consistent character
+- experiment with different voices and styles