task-summary-extractor 9.3.1 → 9.4.0

package/ARCHITECTURE.md

@@ -1,8 +1,7 @@
 # Architecture & Technical Deep Dive
 
 > Internal reference for the pipeline's architecture, processing flows, and design decisions.  
-> For setup instructions, see [README.md](README.md) · [Quick Start](QUICK_START.md)  
-> For module map and roadmap, see [EXPLORATION.md](EXPLORATION.md)
+> For setup instructions, see [README.md](README.md) · [Quick Start](QUICK_START.md)
 
 ---
 
@@ -126,6 +125,7 @@ flowchart TB
 | 1 | **Init** | CLI parsing, interactive folder selection (if no arg), config validation, logger setup, load learning insights, route to dynamic/progress mode |
 | 2 | **Discover** | Find videos/audio, discover documents, resolve user name, check resume state |
 | 3 | **Services** | Firebase auth, Gemini init, prepare document parts |
+| 3.5 | **Deep Summary** | (optional) Pre-summarize context docs with Gemini — 60-80% token savings |
 | 4 | **Process** | Compress → Upload → Analyze → Quality Gate → Retry → Focused Pass |
 | 5 | **Compile** | Cross-segment compilation, diff engine comparison |
 | 6 | **Output** | Write JSON, render Markdown + HTML, upload to Firebase |
@@ -199,7 +199,7 @@ Each video segment goes through this flow (Phase 4 detail):
 
 ```mermaid
 flowchart TB
-    START(["Segment N"]) --> COMPRESS["ffmpeg compress\nH.264 CRF 24, 1.5x speed"]
+    START(["Segment N"]) --> COMPRESS["ffmpeg compress\nH.264 CRF 24, 1.6x speed"]
     COMPRESS --> VERIFY["Verify segment integrity"]
     VERIFY --> UPLOAD_FB["Upload to Firebase Storage\n→ download URL"]
 
@@ -563,7 +563,7 @@ JSONL structured format includes phase spans with timing metrics for observabili
 | **ffmpeg** | System binary | H.264 video compression + segmentation |
 | **Git** | System binary | Change detection for progress tracking |
 
-**Codebase: ~45 files · ~13,000+ lines** · npm package: `task-summary-extractor` · CLI: `taskex`
+**Codebase: ~48 files · ~13,600+ lines** · npm package: `task-summary-extractor` · CLI: `taskex`
 
 ---
 
@@ -634,8 +634,8 @@ The project includes a comprehensive test suite using [vitest](https://vitest.de
 
 | Metric | Value |
 |--------|-------|
-| Test files | 13 |
-| Total tests | 285 |
+| Test files | 15 |
+| Total tests | 331 |
 | Framework | vitest v4.x |
 | Coverage | `@vitest/coverage-v8` |
 
@@ -662,45 +662,45 @@ npm run test:coverage # Coverage report
 |-----|-------------|
 | 📖 [README.md](README.md) | Setup, CLI flags, configuration, features |
 | 📖 [QUICK_START.md](QUICK_START.md) | Step-by-step first-time walkthrough |
-| 🔭 [EXPLORATION.md](EXPLORATION.md) | Module map, line counts, future roadmap |
 
 ---
 
-## JSON Schema Validation
-
-All AI output is validated against JSON Schema definitions in `src/schemas/`:
-
-| Schema | File | Purpose |
-|--------|------|---------|
-| Segment analysis | `analysis-segment.schema.json` | Validates each segment's extracted data |
-| Compiled analysis | `analysis-compiled.schema.json` | Validates the final cross-segment compilation |
+## Deep Summary
 
-Validation is performed by `src/utils/schema-validator.js` using [ajv](https://ajv.js.org/). Validation errors are reported as warnings with contextual hints for the retry/focused-pass cycle — they do not hard-fail the pipeline but are injected as corrective hints when the quality gate triggers a retry.
-
----
+The `--deep-summary` flag (or interactive prompt when many docs are detected) pre-summarizes context documents before segment analysis:
 
-## Test Suite
+```mermaid
+flowchart TB
+    START(["Context Docs"]) --> PARTITION["Partition: summarize vs. keep full"]
+    PARTITION --> SKIP["Skip tiny docs (<500 chars)"]
+    PARTITION --> EXCL["Excluded docs → keep full fidelity"]
+    PARTITION --> TO_SUM["Docs to summarize"]
+    TO_SUM --> TRUNC["Truncate oversized docs (>900K chars)"]
+    TRUNC --> BATCH["Group into batches\n(≤600K chars each)"]
+    BATCH --> AI["Gemini summarization\n(per batch)"]
+    AI --> REPLACE["Replace full content\nwith condensed summaries"]
+    REPLACE --> OUT(["Token-efficient\ncontext docs"])
+```
 
-The project includes a comprehensive test suite using [vitest](https://vitest.dev/):
+| Constant | Value | Purpose |
+|----------|-------|---------|
+| `BATCH_MAX_CHARS` | 600,000 | Max input chars per summarization batch |
+| `MAX_DOC_CHARS` | 900,000 | Hard cap per-document before truncation |
+| `SUMMARY_MAX_OUTPUT` | 16,384 | Max output tokens per summarization call |
+| `MIN_SUMMARIZE_LENGTH` | 500 | Docs below this skip summarization |
 
-| Metric | Value |
-|--------|-------|
-| Test files | 13 |
-| Total tests | 285 |
-| Framework | vitest v4.x |
-| Coverage | `@vitest/coverage-v8` |
+Typical savings: 60-80% reduction in per-segment context tokens. The user can exclude specific docs from summarization via `--exclude-docs` or the interactive picker.
 
-**Test categories:**
+---
 
-| Directory | What's Tested |
-|-----------|---------------|
-| `tests/utils/` | Utility modules: adaptive-budget, cli, confidence-filter, context-manager, diff-engine, format, json-parser, progress-bar, quality-gate, retry, schema-validator |
-| `tests/renderers/` | Renderer modules: html, markdown |
+## Context Window Safety
 
-**Commands:**
+Safeguards to prevent context window overflow:
 
-```bash
-npm test              # Run all tests
-npm run test:watch    # Watch mode
-npm run test:coverage # Coverage report
-```
+| Safeguard | Where | What It Does |
+|-----------|-------|-------------|
+| **P0/P1 hard cap** | `context-manager.js` | Critical docs can't exceed 2× the token budget |
+| **VTT fallback cap** | `context-manager.js` | Full VTT fallback capped at 500K chars |
+| **Doc truncation** | `deep-summary.js` | Oversized docs truncated to 900K chars before summarization |
+| **Compilation pre-flight** | `gemini.js` | Estimates tokens before compilation; trims middle segments if >80% of context |
+| **RESOURCE_EXHAUSTED recovery** | `gemini.js` | On quota/context errors: waits 30s, sheds docs, retries with reduced input |

package/QUICK_START.md

@@ -223,6 +223,7 @@ my-project/runs/{timestamp}/
 | **Force Gemini File API** | `taskex --no-storage-url "my-meeting"` |
 | **Preview without running** | `taskex --dry-run "my-meeting"` |
 | **Deep dive docs** | `taskex --deep-dive "my-meeting"` |
+| **Pre-summarize docs** | `taskex --deep-summary "my-meeting"` |
 | **Generate docs (no video)** | `taskex --dynamic "my-project"` |
 | **Track progress via git** | `taskex --update-progress --repo "C:\project" "my-meeting"` |
 | **Debug mode** | `taskex --log-level debug "my-meeting"` |
@@ -272,4 +273,3 @@ Your recordings, `.env`, logs — everything local is `.gitignore`d and safe.
 |------|-------|
 | Full feature list, all CLI flags, configuration | [README.md](README.md) |
 | How the pipeline works internally | [ARCHITECTURE.md](ARCHITECTURE.md) |
-| Module map, line counts, roadmap | [EXPLORATION.md](EXPLORATION.md) |

package/README.md

@@ -1,13 +1,13 @@
 # Task Summary Extractor
 
-> **v9.0.0** — AI-powered content analysis CLI — meetings, recordings, documents, or any mix. Install globally, run anywhere.
+> **v9.4.0** — AI-powered content analysis CLI — meetings, recordings, documents, or any mix. Install globally, run anywhere.
 
 <p align="center">
   <img src="https://img.shields.io/badge/node-%3E%3D18.0.0-green" alt="Node.js" />
   <img src="https://img.shields.io/badge/gemini-2.5--flash-blue" alt="Gemini" />
   <img src="https://img.shields.io/badge/firebase-11.x-orange" alt="Firebase" />
-  <img src="https://img.shields.io/badge/version-9.0.0-brightgreen" alt="Version" />
-  <img src="https://img.shields.io/badge/tests-285%20passing-brightgreen" alt="Tests" />
+  <img src="https://img.shields.io/badge/version-9.4.0-brightgreen" alt="Version" />
+  <img src="https://img.shields.io/badge/tests-331%20passing-brightgreen" alt="Tests" />
   <img src="https://img.shields.io/badge/npm-task--summary--extractor-red" alt="npm" />
 </p>
 
@@ -62,6 +62,20 @@ taskex --update-progress --repo "C:\my-project" "my-meeting"
 
 > **v7.2.3**: If the call folder isn't a git repo, the tool auto-initializes one for baseline tracking.
 
+### ⚡ Deep Summary (`--deep-summary`)
+
+Pre-summarize context documents to reduce per-segment token usage by 60-80%:
+
+```bash
+taskex --deep-summary --name "Jane" "my-meeting"
+```
+
+Exclude specific docs from summarization (keep at full fidelity):
+
+```bash
+taskex --deep-summary --exclude-docs "code-map.md,sprint.md" "my-meeting"
+```
+
 > See all modes explained with diagrams → [ARCHITECTURE.md](ARCHITECTURE.md#pipeline-phases)
 
 ---
@@ -172,6 +186,8 @@ These are the ones you'll actually use:
 | `--format <type>` | Output format: `md`, `html`, `json`, `pdf`, `docx`, `all` (default: `md`) | `--format html` |
 | `--min-confidence <level>` | Filter items by confidence: `high`, `medium`, `low` | `--min-confidence high` |
 | `--no-html` | Suppress HTML report generation | `--no-html` |
+| `--deep-summary` | Pre-summarize context docs (60-80% token savings) | `--deep-summary` |
+| `--exclude-docs <list>` | Docs to keep full during deep-summary (comma-separated) | `--exclude-docs "code-map.md"` |
 
 **Typical usage:**
 
@@ -198,6 +214,7 @@ Choose what the tool does. Only use one at a time:
 | *(none)* | **Content analysis** | `results.md` + `results.html` — structured task document |
 | `--dynamic` | **Doc generation** | `INDEX.md` + 3–15 topic documents |
 | `--deep-dive` | **Topic explainers** | `INDEX.md` + per-topic deep-dive docs |
+| `--deep-summary` | **Token-efficient analysis** | Same as content analysis, but context docs pre-summarized (60-80% savings) |
 | `--update-progress` | **Progress check** | `progress.md` — item status via git |
 
 **Dynamic mode** also uses:
@@ -259,7 +276,7 @@ taskex [flags] [folder]
 
 CONFIG     --gemini-key  --firebase-key  --firebase-project
            --firebase-bucket  --firebase-domain
-MODES      --dynamic  --deep-dive  --update-progress
+MODES      --dynamic  --deep-dive  --deep-summary  --update-progress
 CORE       --name  --model  --skip-upload  --resume  --reanalyze  --dry-run
 OUTPUT     --format <md|html|json|pdf|docx|all>  --min-confidence <high|medium|low>
            --no-html
@@ -394,7 +411,7 @@ GEMINI_API_KEY=your-key-here
 
 # Optional — uncomment to customize
 # GEMINI_MODEL=gemini-2.5-flash
-# VIDEO_SPEED=1.5
+# VIDEO_SPEED=1.6
 # THINKING_BUDGET=24576
 # LOG_LEVEL=info
 
@@ -413,7 +430,7 @@ GEMINI_API_KEY=your-key-here
 
 | Feature | Description |
 |---------|-------------|
-| **Video/Audio Compression** | H.264 CRF 24, text-optimized sharpening, configurable speed |
+| **Video/Audio Compression** | H.264 CRF 24, text-optimized sharpening, 1.6× speed |
 | **Smart Segmentation** | ≤5 min chunks with boundary-aware splitting |
 | **Cross-Segment Continuity** | Ticket IDs, names, and context carry forward |
 | **Document Discovery** | Auto-finds docs in all subfolders |
@@ -434,6 +451,8 @@ GEMINI_API_KEY=your-key-here
 | **HTML Report** | Self-contained HTML report with collapsible sections, filtering, dark mode |
 | **JSON Schema Validation** | Validates AI output against JSON Schema (segment + compiled) |
 | **Confidence Filter** | `--min-confidence` flag to exclude low-confidence items from output |
+| **Deep Summary** | `--deep-summary` pre-summarizes context docs, 60-80% token savings per segment |
+| **Context Window Safety** | Auto-truncation, pre-flight token checks, RESOURCE_EXHAUSTED recovery |
 | **Multi-Format Output** | `--format` flag: Markdown, HTML, JSON, PDF, DOCX, or all formats at once |
 | **Interactive CLI** | Run with no args → guided experience |
 | **Resume / Checkpoint** | `--resume` continues interrupted runs |
@@ -507,6 +526,7 @@ task-summary-extractor/
 │   │   ├── git.js              Git CLI wrapper
 │   │   └── doc-parser.js       Document text extraction (DOCX, XLSX, PPTX, etc.)
 │   ├── modes/                  AI-heavy pipeline phase modules
+│   │   ├── deep-summary.js     Pre-summarize context docs (deep-summary feature)
 │   │   ├── deep-dive.js        Topic discovery & deep-dive doc generation
 │   │   ├── dynamic-mode.js     Dynamic document planning & generation
 │   │   ├── focused-reanalysis.js  Targeted reanalysis of weak segments
@@ -528,17 +548,14 @@ task-summary-extractor/
 │       ├── schema-validator.js JSON Schema validation (ajv)
 │       └── ... (15 more utility modules)
 │
-├── tests/                      Test suite — 285 tests across 13 files (vitest)
+├── tests/                      Test suite — 331 tests across 15 files (vitest)
 │   ├── utils/                  Utility module tests
 │   └── renderers/              Renderer tests
 │
 ├── QUICK_START.md              Step-by-step setup guide
-├── ARCHITECTURE.md             Technical deep dive
-└── EXPLORATION.md              Roadmap & future features
+└── ARCHITECTURE.md             Technical deep dive
 ```
 
-> Full module map with line counts → [EXPLORATION.md](EXPLORATION.md#full-module-map)
-
 ---
 
 ## npm Scripts
@@ -551,7 +568,7 @@ task-summary-extractor/
 | `npm run check` | Validate environment |
 | `npm start` | Run the pipeline |
 | `npm run help` | Show CLI help |
-| `npm test` | Run test suite (285 tests) |
+| `npm test` | Run test suite (331 tests) |
 | `npm run test:watch` | Run tests in watch mode |
 | `npm run test:coverage` | Run tests with coverage report |
 
@@ -561,6 +578,9 @@ task-summary-extractor/
 
 | Version | Highlights |
 |---------|-----------|
+| **v9.4.0** | **Context window safety** — pre-flight token checks, auto-truncation for oversized docs/VTTs, RESOURCE_EXHAUSTED recovery with automatic doc shedding, chunked compilation for large segment sets, P0/P1 hard cap (2× budget) prevents context overflow, improved deep-summary prompt quality |
+| **v9.3.1** | **Audit & polish** — VIDEO_SPEED 1.5→1.6, `--exclude-docs` flag for non-interactive deep-summary exclusion, friendlier Gemini error messages, dead code removal, DRY RUN_PRESETS |
+| **v9.3.0** | **Deep summary** — `--deep-summary` pre-summarizes context documents (60-80% token savings), interactive doc picker, `--exclude-docs` for CLI automation, batch processing |
 | **v9.0.0** | **CLI UX upgrade** — colors & progress bar, HTML reports, PDF & DOCX output (via puppeteer and docx npm package), JSON Schema validation, confidence filter (`--min-confidence`), pipeline decomposition (`src/phases/` — 9 modules), test suite (285 tests via vitest), multi-format output (`--format`: md/html/json/pdf/docx/all), doc-parser service, shared renderer utilities |
 | **v8.3.0** | **Universal content analysis** — prompt v4.0.0 supports video, audio, documents, and mixed content; input type auto-detection; timestamps conditional on content type; gemini.js bridge text generalized; all markdown docs updated |
 | **v8.2.0** | **Architecture cleanup** — `src/modes/` for AI pipeline phases, `retry.js` self-contained defaults, dead code removal, export trimming, `process_and_upload.js` slim shim, `progress.js` → `checkpoint.js`, merged `prompt.js` into `cli.js` |
@@ -587,7 +607,6 @@ task-summary-extractor/
 |-----|-------------|-------------|
 | 📖 **[QUICK_START.md](QUICK_START.md)** | Full setup walkthrough, examples, troubleshooting | First time using the tool |
 | 🏗️ **[ARCHITECTURE.md](ARCHITECTURE.md)** | Pipeline phases, algorithms, Mermaid diagrams | Understanding how it works |
-| 🔭 **[EXPLORATION.md](EXPLORATION.md)** | Module map, line counts, future roadmap | Contributing or extending |
 
 ---
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "task-summary-extractor",
-  "version": "9.3.1",
+  "version": "9.4.0",
   "description": "AI-powered meeting analysis & document generation CLI — video + document processing, deep dive docs, dynamic mode, interactive CLI with model selection, confidence scoring, learning loop, git progress tracking",
   "main": "process_and_upload.js",
   "bin": {

package/src/modes/deep-summary.js

@@ -36,6 +36,13 @@ const BATCH_MAX_CHARS = 600000;
 /** Minimum content length (chars) to bother summarizing — below this, keep full */
 const MIN_SUMMARIZE_LENGTH = 500;
 
+/**
+ * Hard cap per-document chars before sending to Gemini.
+ * Gemini context = 1M tokens; prompt overhead ~50K tokens; at 0.3 tok/char
+ * 900K chars ≈ 270K tokens — safe with prompt + thinking overhead.
+ */
+const MAX_DOC_CHARS = 900000;
+
 // ======================== BATCH BUILDER ========================
 
 /**
@@ -51,8 +58,22 @@ function buildBatches(docs, maxChars = BATCH_MAX_CHARS) {
   let currentBatch = [];
   let currentChars = 0;
 
-  for (const doc of docs) {
-    const docChars = doc.content ? doc.content.length : 0;
+  for (let doc of docs) {
+    let docChars = doc.content ? doc.content.length : 0;
+
+    // Truncate extremely large docs to avoid exceeding the context window.
+    // Any single doc beyond MAX_DOC_CHARS is capped (tail is dropped) and a
+    // warning is prepended so the summariser knows the content is incomplete.
+    if (docChars > MAX_DOC_CHARS) {
+      const truncated = doc.content.substring(0, MAX_DOC_CHARS);
+      doc = {
+        ...doc,
+        content: `[TRUNCATED — original ${(docChars / 1024).toFixed(0)} KB exceeded the ${(MAX_DOC_CHARS / 1024).toFixed(0)} KB limit; only the first ${(MAX_DOC_CHARS / 1024).toFixed(0)} KB is included]\n\n${truncated}`,
+        _truncatedFrom: docChars,
+      };
+      docChars = doc.content.length;
+      console.warn(`    ${c.warn(`${doc.fileName} truncated from ${(doc._truncatedFrom / 1024).toFixed(0)} KB to ${(MAX_DOC_CHARS / 1024).toFixed(0)} KB for deep summary`)}`);
+    }
 
     // If this single doc exceeds the batch limit, it gets its own batch
     if (docChars > maxChars) {
@@ -120,23 +141,35 @@ async function summarizeBatch(ai, docs, opts = {}) {
 
   const promptText = `You are a precision document summarizer for a meeting analysis pipeline.
 
-Your job: read ALL documents below and produce a CONDENSED version of each that preserves:
-- Every ticket ID, task ID, CR number, or reference number (verbatim)
-- All assignees, reviewers, and responsible parties
-- All statuses (open, closed, in_progress, blocked, etc.)
-- All action items and their owners
-- All blockers, dependencies, and deadlines
-- Key decisions and their rationale
-- File paths and code references
-- Numerical data (percentages, counts, dates, versions)
-
-What to remove:
+Your job: read ALL documents below and produce a CONDENSED version of each that preserves every piece of actionable information.
+
+WHAT TO PRESERVE (in order of importance):
+1. IDENTIFIERS — Every ticket ID, task ID, CR number, PR number, JIRA key, GitHub issue, reference number, version number. Copy these VERBATIM — do not paraphrase or abbreviate IDs.
+2. PEOPLE — All assignees, reviewers, approvers, requesters, and responsible parties. Use full names exactly as they appear.
+3. STATUSES & STATES — All statuses (open, closed, in_progress, blocked, deferred, etc.) and state markers (✅, ⬜, ⏸️, 🔲). Preserve the exact status vocabulary used in the document.
+4. ACTION ITEMS — Every action item, commitment, and deliverable with its owner, deadline, and dependency chain.
+5. BLOCKERS & DEPENDENCIES — What is blocked, by whom, what it blocks downstream.
+6. DECISIONS & RATIONALE — Key decisions and WHY they were made (not just what).
+7. CROSS-REFERENCES — When Document A references something from Document B, preserve that linkage. If ticket X is mentioned in a code-map entry, keep both the ticket ID and the code-map path.
+8. TECHNICAL SPECIFICS — File paths, code references, API endpoints, database tables, configuration keys, environment names (dev/staging/prod).
+9. NUMERICAL DATA — Percentages, counts, dates, deadlines, version numbers, sizes.
+10. CHECKLISTS & PROGRESS — Preserve checklist items with their completion status markers. Include progress ratios (e.g., "35/74 done, 6 blocked").
+
+WHAT TO REMOVE:
 - Verbose explanations of well-known concepts
-- Redundant phrasing and filler text
-- Formatting-only content (decorative headers, dividers)
-- Boilerplate/template text that adds no information
+- Redundant phrasing, filler text, throat-clearing sentences
+- Formatting-only content (decorative headers, horizontal rules, empty sections)
+- Boilerplate/template text that adds no project-specific information
+- Repeated definitions or glossary entries that don't change across documents
 ${focusSection}
 
+QUALITY REQUIREMENTS:
+- Aim for 70-80% size reduction while preserving ALL actionable information.
+- Every ID, every name, every status MUST survive the summarization.
+- If two documents reference the same entity (ticket, file, person), ensure the summary preserves enough context in BOTH summaries for downstream consumers to make the connection.
+- When a document contains a table, preserve the table structure (header + key rows). Omit empty or low-value rows.
+- When a document has nested structure (subsections, indented lists), preserve the hierarchy — use indentation or numbering.
+
 OUTPUT FORMAT:
 Return valid JSON with this structure:
 {
@@ -151,9 +184,6 @@ Return valid JSON with this structure:
   }
 }
 
-Aim for 70-80% size reduction while preserving ALL actionable information.
-Every ID, every name, every status must survive the summarization.
-
 DOCUMENTS TO SUMMARIZE (${docEntries.length} documents):
 
 ${docEntries.join('\n\n')}`;
@@ -162,7 +192,7 @@ ${docEntries.join('\n\n')}`;
     model: config.GEMINI_MODEL,
     contents: [{ role: 'user', parts: [{ text: promptText }] }],
     config: {
-      systemInstruction: 'You are a lossless information compressor. Preserve every ID, name, status, assignment, and actionable detail. Output valid JSON only.',
+      systemInstruction: 'You are a lossless information compressor specialized in engineering and business documents. Preserve every ID, name, status, assignment, dependency, file path, decision rationale, and actionable detail. Maintain cross-document references (when doc A mentions entity from doc B, keep both sides). Output valid JSON only.',
       maxOutputTokens: SUMMARY_MAX_OUTPUT,
       temperature: 0,
       thinkingConfig: { thinkingBudget },
@@ -372,4 +402,5 @@ module.exports = {
   SUMMARY_MAX_OUTPUT,
   BATCH_MAX_CHARS,
   MIN_SUMMARIZE_LENGTH,
+  MAX_DOC_CHARS,
 };

package/src/pipeline.js

@@ -46,7 +46,7 @@ const phaseDeepDive    = require('./phases/deep-dive');
 // --- Utils (for run orchestration + alt modes) ---
 const { c } = require('./utils/colors');
 const { findDocsRecursive } = require('./utils/fs');
-const { promptUserText, selectDocsToExclude } = require('./utils/cli');
+const { promptUser, promptUserText, selectDocsToExclude } = require('./utils/cli');
 const { createProgressBar } = require('./utils/progress-bar');
 const { buildHealthReport, printHealthDashboard } = require('./utils/health-dashboard');
 const { saveHistory, buildHistoryEntry } = require('./utils/learning-loop');
@@ -96,6 +96,23 @@ async function run() {
   bar.tick('Services ready');
 
   // Phase 3.5 (optional): Deep Summary — pre-summarize context docs
+  // If user didn't pass --deep-summary but has many context docs, offer it interactively
+  if (!fullCtx.opts.deepSummary && process.stdin.isTTY && fullCtx.ai && fullCtx.contextDocs.length >= 3) {
+    const inlineDocs = fullCtx.contextDocs.filter(d => d.type === 'inlineText' && d.content);
+    const totalChars = inlineDocs.reduce((sum, d) => sum + d.content.length, 0);
+    const totalTokensEstimate = Math.ceil(totalChars * 0.3);
+    // Only offer when context is large enough to benefit (>100K tokens)
+    if (totalTokensEstimate > 100000) {
+      console.log('');
+      console.log(`  ${c.cyan('You have')} ${c.highlight(inlineDocs.length)} ${c.cyan('context docs')} (~${c.highlight((totalTokensEstimate / 1000).toFixed(0) + 'K')} ${c.cyan('tokens)')}`);
+      console.log(`  ${c.dim('Deep summary can reduce per-segment context by 60-80%, saving time and cost.')}`);
+      const wantDeepSummary = await promptUser(`  ${c.cyan('Enable deep summary?')} [y/N] `);
+      if (wantDeepSummary) {
+        fullCtx.opts.deepSummary = true;
+      }
+    }
+  }
+
   if (fullCtx.opts.deepSummary && fullCtx.ai && fullCtx.contextDocs.length > 0) {
     // Interactive picker: let user choose docs to keep at full fidelity
     if (process.stdin.isTTY && fullCtx.opts.deepSummaryExclude.length === 0) {

package/src/services/gemini.js

@@ -459,16 +459,53 @@ async function processWithGemini(ai, filePath, displayName, contextDocs = [], pr
         throw reuploadErr;
       }
     } else {
-      // Log request diagnostics for other errors to aid debugging
-      const partSummary = contentParts.map((p, i) => {
-        if (p.fileData) return `  [${i}] fileData: ${p.fileData.mimeType} → ${(p.fileData.fileUri || '').substring(0, 120)}`;
-        if (p.text) return `  [${i}] text: ${p.text.length} chars → ${p.text.substring(0, 80).replace(/\n/g, ' ')}...`;
-        return `  [${i}] unknown part`;
-      });
-      console.error(`    ${c.error('Request diagnostics:')}`);
-      console.error(`    Model: ${config.GEMINI_MODEL} | Parts: ${contentParts.length} | maxOutput: 65536`);
-      partSummary.forEach(s => console.error(`    ${s}`));
-      throw apiErr;
+      // Handle RESOURCE_EXHAUSTED specifically — shed lower-priority docs and retry
+      if (errMsg.includes('RESOURCE_EXHAUSTED') || errMsg.includes('429') || errMsg.includes('quota')) {
+        console.warn(`    ${c.warn('Context window or quota exceeded — shedding docs and retrying after 30s...')}`);
+        await new Promise(r => setTimeout(r, 30000));
+        // Rebuild with half the doc budget
+        const reducedBudget = Math.floor(docBudget * 0.5);
+        const { selected: reducedDocs } = selectDocsByBudget(contextDocs, reducedBudget, { segmentIndex });
+        const reducedParts = [contentParts[0]]; // keep video
+        for (const doc of reducedDocs) {
+          if (doc.type === 'inlineText') {
+            let content = doc.content;
+            const isVtt = doc.fileName.toLowerCase().endsWith('.vtt') || doc.fileName.toLowerCase().endsWith('.srt');
+            if (isVtt && segmentStartSec != null && segmentEndSec != null) {
+              content = sliceVttForSegment(content, segmentStartSec, segmentEndSec);
+            }
+            reducedParts.push({ text: `=== Document: ${doc.fileName} ===\n${content}` });
+          } else if (doc.type === 'fileData') {
+            reducedParts.push({ fileData: { mimeType: doc.mimeType, fileUri: doc.fileUri } });
+          }
+        }
+        // Re-add prompt/context parts (last 3-5 parts are prompt, focus, etc.)
+        const nonDocParts = contentParts.slice(1 + selectedDocs.length);
+        reducedParts.push(...nonDocParts);
+        requestPayload.contents[0].parts = reducedParts;
+        console.log(`    Reduced to ${reducedDocs.length} docs (budget: ${(reducedBudget / 1000).toFixed(0)}K tokens)`);
+        try {
+          response = await withRetry(
+            () => ai.models.generateContent(requestPayload),
+            { label: `Gemini segment analysis — reduced docs (${displayName})`, maxRetries: 1, baseDelay: 5000 }
+          );
+          console.log(`    ${c.success('Reduced-context retry succeeded')}`);
+        } catch (reduceErr) {
+          console.error(`    ${c.error(`Reduced-context retry also failed: ${reduceErr.message}`)}`);
+          throw reduceErr;
+        }
+      } else {
+        // Log request diagnostics for other errors to aid debugging
+        const partSummary = contentParts.map((p, i) => {
+          if (p.fileData) return `  [${i}] fileData: ${p.fileData.mimeType} → ${(p.fileData.fileUri || '').substring(0, 120)}`;
+          if (p.text) return `  [${i}] text: ${p.text.length} chars → ${p.text.substring(0, 80).replace(/\n/g, ' ')}...`;
+          return `  [${i}] unknown part`;
+        });
+        console.error(`    ${c.error('Request diagnostics:')}`);
+        console.error(`    Model: ${config.GEMINI_MODEL} | Parts: ${contentParts.length} | maxOutput: 65536`);
+        partSummary.forEach(s => console.error(`    ${s}`));
+        throw apiErr;
+      }
     }
   }
   const durationMs = Date.now() - t0;
@@ -628,6 +665,60 @@ ${segmentDumps}`;
 
   const contentParts = [{ text: compilationPrompt }];
 
+  // ------- Pre-flight context window check -------
+  const estimatedInputTokens = estimateTokens(compilationPrompt);
+  const safeLimit = Math.floor(config.GEMINI_CONTEXT_WINDOW * 0.80); // 80% of context window
+  if (estimatedInputTokens > safeLimit) {
+    console.warn(`  ${c.warn(`Compilation input (~${(estimatedInputTokens / 1000).toFixed(0)}K tokens) exceeds 80% of context window (${(safeLimit / 1000).toFixed(0)}K). Trimming older segment detail...`)}`);
+    // Re-build segment dumps with aggressive compression: keep only first & last 2 segments
+    // at full detail, compress the middle ones to IDs + statuses only.
+    const trimmedDumps = allSegmentAnalyses.map((analysis, idx) => {
+      const clean = { ...analysis };
+      delete clean._geminiMeta;
+      delete clean.seg;
+      delete clean.conversation_transcript;
+      const isEdge = idx < 2 || idx >= allSegmentAnalyses.length - 2;
+      if (!isEdge) {
+        // Aggressive compression for middle segments
+        if (clean.tickets) {
+          clean.tickets = clean.tickets.map(t => ({
+            ticket_id: t.ticket_id, status: t.status, title: t.title,
+            assignee: t.assignee, source_segment: t.source_segment,
+          }));
+        }
+        if (clean.change_requests) {
+          clean.change_requests = clean.change_requests.map(cr => ({
+            id: cr.id, status: cr.status, title: cr.title,
+            assigned_to: cr.assigned_to, source_segment: cr.source_segment,
+          }));
+        }
+        if (clean.action_items) {
+          clean.action_items = clean.action_items.map(ai => ({
+            id: ai.id, description: ai.description, assigned_to: ai.assigned_to,
+            status: ai.status, source_segment: ai.source_segment,
+          }));
+        }
+        delete clean.file_references;
+        clean.summary = (clean.summary || '').substring(0, 200);
+      } else {
+        if (clean.tickets) {
+          clean.tickets = clean.tickets.map(t => {
+            const tc = { ...t };
+            if (tc.comments && tc.comments.length > 5) {
+              tc.comments = tc.comments.slice(0, 5);
+              tc.comments.push({ note: `...${t.comments.length - 5} more comments omitted` });
+            }
+            return tc;
+          });
+        }
+      }
+      return `=== SEGMENT ${idx + 1} OF ${allSegmentAnalyses.length} ===\n${JSON.stringify(clean, null, 2)}`;
+    }).join('\n\n');
+    contentParts[0] = { text: compilationPrompt.replace(segmentDumps, trimmedDumps) };
+    const newEstimate = estimateTokens(contentParts[0].text);
+    console.log(`  Trimmed compilation input to ~${(newEstimate / 1000).toFixed(0)}K tokens`);
+  }
+
   const requestPayload = {
     model: config.GEMINI_MODEL,
     contents: [{ role: 'user', parts: contentParts }],
@@ -640,10 +731,44 @@ ${segmentDumps}`;
 
   const t0 = Date.now();
   console.log(`  Compiling with ${config.GEMINI_MODEL}...`);
-  const response = await withRetry(
-    () => ai.models.generateContent(requestPayload),
-    { label: 'Gemini final compilation', maxRetries: 2, baseDelay: 5000 }
-  );
+  let response;
+  try {
+    response = await withRetry(
+      () => ai.models.generateContent(requestPayload),
+      { label: 'Gemini final compilation', maxRetries: 2, baseDelay: 5000 }
+    );
+  } catch (compileErr) {
+    const errMsg = compileErr.message || '';
+    if (errMsg.includes('RESOURCE_EXHAUSTED') || errMsg.includes('429') || errMsg.includes('quota')) {
+      console.warn(`  ${c.warn('Context window or quota exceeded during compilation — waiting 30s and retrying with reduced input...')}`);
+      await new Promise(r => setTimeout(r, 30000));
+      // Halve the compilation prompt by keeping only edge segments
+      const miniDumps = allSegmentAnalyses.map((analysis, idx) => {
+        const clean = { tickets: (analysis.tickets || []).map(t => ({ ticket_id: t.ticket_id, status: t.status, title: t.title, assignee: t.assignee })),
+          change_requests: (analysis.change_requests || []).map(cr => ({ id: cr.id, status: cr.status, title: cr.title })),
+          action_items: (analysis.action_items || []).map(ai => ({ id: ai.id, description: ai.description, assigned_to: ai.assigned_to, status: ai.status })),
+          blockers: (analysis.blockers || []).map(b => ({ id: b.id, description: b.description, status: b.status })),
+          scope_changes: analysis.scope_changes || [],
+          your_tasks: analysis.your_tasks || {},
+          summary: (analysis.summary || '').substring(0, 300),
+        };
+        return `=== SEGMENT ${idx + 1} OF ${allSegmentAnalyses.length} ===\n${JSON.stringify(clean, null, 2)}`;
+      }).join('\n\n');
+      requestPayload.contents[0].parts = [{ text: compilationPrompt.replace(/SEGMENT ANALYSES:\n[\s\S]*$/, `SEGMENT ANALYSES:\n${miniDumps}`) }];
+      try {
+        response = await withRetry(
+          () => ai.models.generateContent(requestPayload),
+          { label: 'Gemini compilation (reduced)', maxRetries: 1, baseDelay: 5000 }
+        );
+        console.log(`  ${c.success('Reduced compilation succeeded')}`);
+      } catch (reduceErr) {
+        console.error(`  ${c.error(`Reduced compilation also failed: ${reduceErr.message}`)}`);
+        throw reduceErr;
+      }
+    } else {
+      throw compileErr;
+    }
+  }
   const durationMs = Date.now() - t0;
   const rawText = response.text;
 

package/src/utils/context-manager.js

@@ -29,6 +29,14 @@ function estimateDocTokens(doc) {
   return 500;
 }
 
+/**
+ * Hard character limit for VTT fallback.
+ * When VTT parsing fails (0 cues), the full VTT is returned.
+ * Cap it so a huge transcript can't blow the context window.
+ * 500K chars ≈ 150K tokens — leaves plenty of room for docs + prompt.
+ */
+const VTT_FALLBACK_MAX_CHARS = 500000;
+
 // ════════════════════════════════════════════════════════════
 //  Priority Classification
 // ════════════════════════════════════════════════════════════
@@ -100,12 +108,16 @@ function selectDocsByBudget(allDocs, tokenBudget, opts = {}) {
   const excluded = [];
   let usedTokens = 0;
 
+  // Hard cap: even P0/P1 docs may not exceed 2× the budget.
+  // This prevents a handful of huge critical docs from blowing the context window.
+  const hardCap = tokenBudget * 2;
+
   for (const item of classified) {
     if (usedTokens + item.tokens <= tokenBudget) {
       selected.push(item.doc);
       usedTokens += item.tokens;
-    } else if (item.priority <= PRIORITY.HIGH) {
-      // P0 and P1 are always included even if over budget
+    } else if (item.priority <= PRIORITY.HIGH && usedTokens + item.tokens <= hardCap) {
+      // P0 and P1 are always included even if over budget, up to the hard cap
       selected.push(item.doc);
       usedTokens += item.tokens;
     } else {
@@ -171,14 +183,28 @@ function parseVttCues(vttContent) {
  */
 function sliceVttForSegment(vttContent, segStartSec, segEndSec, overlapSec = 30) {
   const cues = parseVttCues(vttContent);
-  if (cues.length === 0) return vttContent; // fallback: return full VTT
+  if (cues.length === 0) {
+    // Fallback: return full VTT but cap size to avoid context window overflow
+    if (vttContent.length > VTT_FALLBACK_MAX_CHARS) {
+      return vttContent.substring(0, VTT_FALLBACK_MAX_CHARS) +
+        `\n\n[TRUNCATED — original VTT was ${(vttContent.length / 1024).toFixed(0)} KB; capped at ${(VTT_FALLBACK_MAX_CHARS / 1024).toFixed(0)} KB]`;
+    }
+    return vttContent;
+  }
 
   const rangeStart = Math.max(0, segStartSec - overlapSec);
   const rangeEnd = segEndSec + overlapSec;
 
   const filtered = cues.filter(c => c.endSec >= rangeStart && c.startSec <= rangeEnd);
 
-  if (filtered.length === 0) return vttContent; // fallback
+  if (filtered.length === 0) {
+    // Fallback with cap
+    if (vttContent.length > VTT_FALLBACK_MAX_CHARS) {
+      return vttContent.substring(0, VTT_FALLBACK_MAX_CHARS) +
+        `\n\n[TRUNCATED — original VTT was ${(vttContent.length / 1024).toFixed(0)} KB; capped at ${(VTT_FALLBACK_MAX_CHARS / 1024).toFixed(0)} KB]`;
+    }
+    return vttContent;
+  }
 
   const header = `WEBVTT\n\n[Segment transcript: ${formatHMS(segStartSec)} — ${formatHMS(segEndSec)}]\n[Showing cues from ${formatHMS(rangeStart)} to ${formatHMS(rangeEnd)} with ${overlapSec}s overlap]\n`;
 
@@ -492,4 +518,5 @@ module.exports = {
   buildProgressiveContext,
   buildSegmentFocus,
   detectBoundaryContext,
+  VTT_FALLBACK_MAX_CHARS,
 };