videonut 1.0.1 → 1.1.0

package/config.yaml

@@ -0,0 +1,62 @@
+# VideoNut Configuration
+# This file is managed by Topic Scout agent. All other agents READ ONLY.
+
+# ═══════════════════════════════════════════════════════════════════
+# USER SETTINGS
+# ═══════════════════════════════════════════════════════════════════
+user_name: "Producer"
+communication_language: "Telugu"
+
+# ═══════════════════════════════════════════════════════════════════
+# PROJECT SETTINGS (Set by Topic Scout)
+# ═══════════════════════════════════════════════════════════════════
+projects_folder: "./Projects"
+current_project: "gemini_2025-12-30_SEBI-Hindenburg_004"
+
+# ═══════════════════════════════════════════════════════════════════
+# VIDEO PRODUCTION SETTINGS (Set by Topic Scout)
+# ═══════════════════════════════════════════════════════════════════
+video_format: "Investigative Documentary"
+target_duration: 20
+target_line_count: 2700
+audio_language: "Telugu"
+
+# ═══════════════════════════════════════════════════════════════════
+# SCOPE & REGION (Set by Topic Scout - User Selected)
+# ═══════════════════════════════════════════════════════════════════
+# scope: international | national | regional
+scope: "national"
+
+# country: Only set if scope is "national" (e.g., India, USA, UK)
+country: "India"
+
+# region: User selected region (NOT auto-derived from language)
+# Examples: "Telangana", "Andhra Pradesh", "Maharashtra", "Tamil Nadu", "Pan-India"
+region: "Pan-India"
+
+# ═══════════════════════════════════════════════════════════════════
+# INDUSTRY TAG (Set by Topic Scout)
+# ═══════════════════════════════════════════════════════════════════
+# Helps agents stay in context and use appropriate sources
+# Options: Finance, Stock Market, Political, Crime, Social Awareness, 
+#          Technology, Entertainment, Sports, Health, Environment, Business, Other
+industry_tag: "Political"
+
+# Industry-specific sources (auto-populated based on industry_tag)
+# Finance: RBI, SEBI, Economic Times, Mint
+# Stock Market: NSE, BSE, MoneyControl, TradingView
+# Political: Election Commission, PRS Legislative, Parliament videos
+# Crime: Court records, Police statements, NCRB data
+# Social Awareness: NGO reports, Government schemes, RTI data
+
+# ═══════════════════════════════════════════════════════════════════
+# EIC CORRECTION TRACKING (Set by EIC after review)
+# ═══════════════════════════════════════════════════════════════════
+# correction_log: Path to the correction log file (relative to project folder)
+# Status: pending_review | corrections_needed | approved
+correction_log: ""
+correction_status: "pending_review"
+
+# Agents with pending corrections (comma-separated)
+# Example: "investigator,scriptwriter,director"
+agents_with_errors: ""

package/docs/AUDIT_REPORT.md

@@ -0,0 +1,364 @@
+# VideoNut System Audit Report
+
+**Date:** 2026-01-04
+**Purpose:** Verify the complete asset workflow works like a human would
+
+---
+
+## 🎯 AUDIT SCOPE
+
+Testing if the system can:
+1. Find YouTube video → Get captions → Find timestamp → Download specific clip
+2. Find PDF → Search for keywords → Screenshot relevant page
+3. Find News Article → Scroll to quote → Highlight and screenshot
+
+---
+
+## ✅ AUDIT RESULTS SUMMARY
+
+| Workflow | Status | Tool Chain |
+|----------|--------|------------|
+| YouTube Clip Extraction | ✅ WORKING | `youtube_search.py` → `caption_reader.py` → `clip_grabber.py` |
+| PDF Search & Screenshot | ✅ WORKING (ENHANCED) | `pdf_reader.py` → `pdf_screenshotter.py` (NEW) |
+| Article Quote Screenshot | ✅ WORKING (FIXED) | `link_checker.py` → `article_screenshotter.py` |
+| Link Validation | ✅ WORKING | `link_checker.py` |
+
+---
+
+## 📹 WORKFLOW 1: YOUTUBE VIDEO CLIP EXTRACTION
+
+### How a Human Would Do It:
+1. Search YouTube for "PM Modi speech on electoral bonds"
+2. Find a video
+3. Watch/skim to find the relevant quote
+4. Note the timestamp (e.g., 5:23 to 5:45)
+5. Download that specific clip
+
+### How Our System Does It:
+
+```
+┌─────────────────────────────────────────────────────────────────────────┐
+│  STEP 1: Director/Investigator searches for videos                     │
+├─────────────────────────────────────────────────────────────────────────┤
+│                                                                         │
+│  COMMAND:                                                               │
+│  python youtube_search.py --query "PM Modi electoral bonds speech" ─►  │
+│                                                                         │
+│  OUTPUT:                                                                │
+│  ┌────────────────────────────────────────────────────────────────────┐│
+│  │ 1. PM Modi addresses Parliament on Electoral Bonds                ││
+│  │    URL: https://youtube.com/watch?v=abc123                        ││
+│  │    Views: 1.2M | Duration: 15:30                                  ││
+│  │                                                                    ││
+│  │ 2. Full Press Conference - Electoral Bonds Explained              ││
+│  │    URL: https://youtube.com/watch?v=def456                        ││
+│  └────────────────────────────────────────────────────────────────────┘│
+│                                                                         │
+└─────────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────────┐
+│  STEP 2: Get transcript and search for quote                           │
+├─────────────────────────────────────────────────────────────────────────┤
+│                                                                         │
+│  COMMAND:                                                               │
+│  python caption_reader.py --url "https://youtube.com/watch?v=abc123"   │
+│                           --search "transparency"                       │
+│                                                                         │
+│  OUTPUT:                                                                │
+│  ┌────────────────────────────────────────────────────────────────────┐│
+│  │ 🔍 Found 3 matches for 'transparency':                             ││
+│  │                                                                    ││
+│  │ [05:23] "Electoral bonds ensure complete transparency"            ││
+│  │ [08:45] "We brought transparency to political funding"            ││
+│  │ [12:10] "Maximum transparency has been achieved"                   ││
+│  │                                                                    ││
+│  │ 📋 Suggested clip range: 05:23 - 12:25                            ││
+│  └────────────────────────────────────────────────────────────────────┘│
+│                                                                         │
+└─────────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────────┐
+│  STEP 3: Get EXACT timestamp for specific quote                        │
+├─────────────────────────────────────────────────────────────────────────┤
+│                                                                         │
+│  COMMAND:                                                               │
+│  python caption_reader.py --url "https://youtube.com/watch?v=abc123"   │
+│                           --find-quote "complete transparency" --json   │
+│                                                                         │
+│  OUTPUT (JSON):                                                         │
+│  ┌────────────────────────────────────────────────────────────────────┐│
+│  │ {                                                                  ││
+│  │   "found": true,                                                   ││
+│  │   "quote": "Electoral bonds ensure complete transparency",        ││
+│  │   "timestamp": "05:23",                                           ││
+│  │   "clip_start": "04:53",    ◄── 30 sec before                     ││
+│  │   "clip_end": "05:58",      ◄── 30 sec after                      ││
+│  │   "context": [...]                                                 ││
+│  │ }                                                                  ││
+│  └────────────────────────────────────────────────────────────────────┘│
+│                                                                         │
+│  ✅ NOW WE KNOW: Download from 04:53 to 05:58                          │
+│                                                                         │
+└─────────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────────┐
+│  STEP 4: Scavenger adds to asset_manifest.md                           │
+├─────────────────────────────────────────────────────────────────────────┤
+│                                                                         │
+│  MANIFEST ENTRY:                                                        │
+│  ┌────────────────────────────────────────────────────────────────────┐│
+│  │ | Scene | URL | Type | Timestamp | Quote |                        ││
+│  │ |-------|-----|------|-----------|-------|                        ││
+│  │ | 5 | youtube.com/watch?v=abc123 | Video Clip | 04:53-05:58 |      ││
+│  │ |   |   |   |   | "Electoral bonds ensure complete transparency" |││
+│  └────────────────────────────────────────────────────────────────────┘│
+│                                                                         │
+└─────────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────────┐
+│  STEP 5: Archivist downloads the EXACT clip                            │
+├─────────────────────────────────────────────────────────────────────────┤
+│                                                                         │
+│  COMMAND:                                                               │
+│  python clip_grabber.py --url "https://youtube.com/watch?v=abc123"     │
+│                         --start "04:53" --end "05:58"                   │
+│                         --output "assets/005_PM_transparency.mp4"       │
+│                                                                         │
+│  OUTPUT:                                                                │
+│  ┌────────────────────────────────────────────────────────────────────┐│
+│  │ ✅ Download successful.                                            ││
+│  │ ✅ File validation: 005_PM_transparency.mp4 (2.3 MB)              ││
+│  └────────────────────────────────────────────────────────────────────┘│
+│                                                                         │
+│  Result: 65 second clip (4:53 to 5:58) downloaded, NOT the full 15min!│
+│                                                                         │
+└─────────────────────────────────────────────────────────────────────────┘
+```
+
+### What If Captions Are Not Available?
+
+```
+┌─────────────────────────────────────────────────────────────────────────┐
+│  FALLBACK: No Captions Available                                        │
+├─────────────────────────────────────────────────────────────────────────┤
+│                                                                         │
+│  caption_reader.py OUTPUT:                                              │
+│  "Error: No captions available for this video"                          │
+│                                                                         │
+│  ARCHIVIST ACTION:                                                      │
+│  1. Download 30-second preview: --start "00:00" --end "00:30"          │
+│  2. Log to MANUAL_REQUIRED.txt:                                        │
+│     "Scene 5: No captions available. Downloaded preview.               │
+│      Human must watch and find correct timestamp."                     │
+│                                                                         │
+│  ✅ System doesn't fail - gracefully degrades to manual review         │
+│                                                                         │
+└─────────────────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## 📄 WORKFLOW 2: PDF SEARCH & SCREENSHOT
+
+### How a Human Would Do It:
+1. Download or open PDF (e.g., Supreme Court judgment)
+2. Press Ctrl+F, search for "electoral bonds"
+3. Go to the page where it's found
+4. Take screenshot of that page
+
+### How Our System Does It (ENHANCED):
+
+```
+┌─────────────────────────────────────────────────────────────────────────┐
+│  STEP 1: Search for keyword in PDF                                     │
+├─────────────────────────────────────────────────────────────────────────┤
+│                                                                         │
+│  COMMAND:                                                               │
+│  python pdf_reader.py --url "https://sci.gov.in/judgment.pdf"          │
+│                       --search "electoral bonds"                        │
+│                                                                         │
+│  OUTPUT:                                                                │
+│  ┌────────────────────────────────────────────────────────────────────┐│
+│  │ 📄 PDF loaded: 156 pages                                           ││
+│  │ 🔍 Searching for: 'electoral bonds'                                ││
+│  │                                                                    ││
+│  │ ✅ Found 8 matches:                                                ││
+│  │                                                                    ││
+│  │ ════════════════════════════════════════════════════════════       ││
+│  │ 📍 Match 1 - Page 23                                               ││
+│  │ ════════════════════════════════════════════════════════════       ││
+│  │ Line: The electoral bonds scheme was introduced in 2017...         ││
+│  │                                                                    ││
+│  │ Context:                                                           ││
+│  │ The Finance Act of 2017 made several amendments...                 ││
+│  │ The electoral bonds scheme was introduced in 2017...               ││
+│  │ These bonds could only be purchased from SBI...                    ││
+│  │                                                                    ││
+│  │ 📸 Suggested page for screenshot: Page 23                          ││
+│  └────────────────────────────────────────────────────────────────────┘│
+│                                                                         │
+└─────────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────────┐
+│  STEP 2: Screenshot that specific page (NEW TOOL)                      │
+├─────────────────────────────────────────────────────────────────────────┤
+│                                                                         │
+│  COMMAND:                                                               │
+│  python pdf_screenshotter.py --url "https://sci.gov.in/judgment.pdf"   │
+│                              --search "electoral bonds"                 │
+│                              --output "assets/sc_judgment_page23.png"   │
+│                                                                         │
+│  OUTPUT:                                                                │
+│  ┌────────────────────────────────────────────────────────────────────┐│
+│  │ 📥 Downloading PDF...                                              ││
+│  │ ✅ PDF downloaded                                                  ││
+│  │ 🔍 Searching for 'electoral bonds' in PDF...                       ││
+│  │ ✅ Found 'electoral bonds' on page 23                              ││
+│  │ 🌐 Opening PDF in browser...                                       ││
+│  │ 📸 Taking screenshot of page 23...                                 ││
+│  │ ✅ Screenshot saved: assets/sc_judgment_page23.png                 ││
+│  └────────────────────────────────────────────────────────────────────┘│
+│                                                                         │
+└─────────────────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## 📰 WORKFLOW 3: NEWS ARTICLE SCREENSHOT
+
+### How a Human Would Do It:
+1. Open the news article
+2. Scroll down to find the quote "PM Modi said..."
+3. Take screenshot with that quote visible
+
+### How Our System Does It:
+
+```
+┌─────────────────────────────────────────────────────────────────────────┐
+│  STEP 1: Director identifies the quote to capture                      │
+├─────────────────────────────────────────────────────────────────────────┤
+│                                                                         │
+│  In master_script.md:                                                   │
+│  [Source: https://timesofindia.com/article/123]                        │
+│  [Screenshot-Quote: "₹560 crore donated after ED raid"]                │
+│                                                                         │
+└─────────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────────┐
+│  STEP 2: Archivist takes screenshot with highlight                      │
+├─────────────────────────────────────────────────────────────────────────┤
+│                                                                         │
+│  COMMAND:                                                               │
+│  python article_screenshotter.py                                        │
+│         --url "https://timesofindia.com/article/123"                    │
+│         --quote "₹560 crore donated after ED raid"                      │
+│         --output "assets/007_quid_pro_quo.png"                          │
+│                                                                         │
+│  PROCESS (INSIDE THE TOOL):                                             │
+│  ┌────────────────────────────────────────────────────────────────────┐│
+│  │ 1. 🌐 Open article URL                                             ││
+│  │ 2. ⏳ Wait for page to fully load (networkidle)                    ││
+│  │ 3. 🧹 Close cookie popups                                          ││
+│  │ 4. 🔍 Search for quote using 3 strategies:                         ││
+│  │    - Strategy 1: Playwright exact match                           ││
+│  │    - Strategy 2: First 5 words match                               ││
+│  │    - Strategy 3: JavaScript deep search                            ││
+│  │ 5. ✅ Quote found!                                                  ││
+│  │ 6. 📍 Scroll to CENTER quote in viewport                           ││
+│  │    - Method 1: scrollIntoView({ block: 'center' })                 ││
+│  │    - Method 2: scrollTo() calculation (fallback)                   ││
+│  │    - Method 3: Playwright scroll_into_view_if_needed()             ││
+│  │ 7. 🎨 Highlight with yellow background + orange border             ││
+│  │    - Uses !important to override site CSS                          ││
+│  │ 8. 📸 Take screenshot                                              ││
+│  └────────────────────────────────────────────────────────────────────┘│
+│                                                                         │
+│  OUTPUT:                                                                │
+│  ┌────────────────────────────────────────────────────────────────────┐│
+│  │ 🌐 Navigating to https://timesofindia.com/article/123...           ││
+│  │ ⏳ Waiting for dynamic content to load...                          ││
+│  │ 🧹 Closing popups...                                               ││
+│  │ 🔍 Searching for quote: '₹560 crore donated after ED raid'         ││
+│  │   Strategy 1: Exact text match...                                  ││
+│  │   ✅ Found with Strategy 1                                         ││
+│  │ 📍 Scrolling quote to center of viewport...                        ││
+│  │   ✅ Scroll method 1 (scrollIntoView) succeeded                    ││
+│  │ 🎨 Highlighting quote...                                           ││
+│  │   ✅ Quote highlighted with yellow background + orange border      ││
+│  │ 📸 Taking screenshot...                                            ││
+│  │ ✅ Screenshot saved: assets/007_quid_pro_quo.png (245,320 bytes)   ││
+│  └────────────────────────────────────────────────────────────────────┘│
+│                                                                         │
+│  RESULT: Screenshot shows the exact quote highlighted, not page header!│
+│                                                                         │
+└─────────────────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## 🔗 WORKFLOW 4: LINK VALIDATION
+
+### Every URL is checked before use:
+
+```
+┌─────────────────────────────────────────────────────────────────────────┐
+│  Link Validation in Every Step                                          │
+├─────────────────────────────────────────────────────────────────────────┤
+│                                                                         │
+│  COMMAND:                                                               │
+│  python link_checker.py "https://example.com/article"                  │
+│                                                                         │
+│  POSSIBLE OUTPUTS:                                                      │
+│  ┌────────────────────────────────────────────────────────────────────┐│
+│  │ ✅ VALID: URL is accessible (HTTP 200)                             ││
+│  │ ⚠️ REDIRECT: URL redirects to another location                     ││
+│  │ ❌ INVALID: URL returns 404 or timeout                             ││
+│  │ ⚠️ REQUIRES AUTH: URL needs login                                  ││
+│  └────────────────────────────────────────────────────────────────────┘│
+│                                                                         │
+│  WHO USES IT:                                                           │
+│  - Scavenger: Before adding URL to manifest                            │
+│  - Archivist: Before attempting download                               │
+│  - EIC: To verify all URLs still work                                  │
+│                                                                         │
+└─────────────────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## 📊 TOOL CAPABILITY MATRIX
+
+| Capability | Tool | Status |
+|------------|------|--------|
+| Search YouTube videos | `youtube_search.py` | ✅ Working |
+| Get video captions | `caption_reader.py` | ✅ Working |
+| Search keywords in captions | `caption_reader.py --search` | ✅ Working |
+| Find exact quote timestamp | `caption_reader.py --find-quote` | ✅ Working |
+| Download specific clip | `clip_grabber.py --start --end` | ✅ Working |
+| Validate URLs | `link_checker.py` | ✅ Working |
+| Read PDF content | `pdf_reader.py` | ✅ Working |
+| **Search keywords in PDF** | `pdf_reader.py --search` | ✅ **ENHANCED** |
+| **Screenshot PDF page** | `pdf_screenshotter.py` | ✅ **NEW** |
+| Screenshot article | `article_screenshotter.py` | ✅ Working |
+| **Scroll to quote** | `article_screenshotter.py` | ✅ **FIXED** |
+| **Highlight quote** | `article_screenshotter.py` | ✅ **FIXED** |
+| Download images | `image_grabber.py` | ✅ Working |
+| Archive URLs | `archive_url.py` | ✅ Working |
+
+---
+
+## ✅ CONCLUSION
+
+The system now works like a human would:
+
+1. **YouTube clips**: Searches → Reads captions → Finds timestamp → Downloads exact clip
+2. **PDFs**: Downloads → Searches for keyword → Screenshots the relevant page
+3. **Articles**: Opens → Scrolls to quote → Highlights → Screenshots
+
+All agents are properly using these tools in the correct workflow order.