task-summary-extractor 9.2.2 → 9.4.0

package/.env.example

@@ -12,8 +12,8 @@ GEMINI_API_KEY=your_gemini_api_key
 GEMINI_MODEL=gemini-2.5-flash
 
 # ======================== VIDEO PROCESSING ========================
-# Speed multiplier (default: 1.5)
-VIDEO_SPEED=1.5
+# Speed multiplier (default: 1.6)
+VIDEO_SPEED=1.6
 # Segment duration in seconds (default: 280)
 VIDEO_SEGMENT_TIME=280
 # ffmpeg preset: ultrafast, superfast, veryfast, faster, fast, medium, slow, slower, veryslow
@@ -36,3 +36,7 @@ THINKING_BUDGET=24576
 COMPILATION_THINKING_BUDGET=10240
 # Max polling time for Gemini File API processing in ms (default: 300000 = 5 min)
 GEMINI_POLL_TIMEOUT_MS=300000
+
+# ======================== NPM PUBLISHING ========================
+# Automation token for npm publish (optional — if not set, browser sign-in is used)
+# NPM_TOKEN=npm_your_token_here

package/ARCHITECTURE.md

@@ -1,8 +1,7 @@
 # Architecture & Technical Deep Dive
 
 > Internal reference for the pipeline's architecture, processing flows, and design decisions.  
-> For setup instructions, see [README.md](README.md) · [Quick Start](QUICK_START.md)  
-> For module map and roadmap, see [EXPLORATION.md](EXPLORATION.md)
+> For setup instructions, see [README.md](README.md) · [Quick Start](QUICK_START.md)
 
 ---
 
@@ -126,6 +125,7 @@ flowchart TB
 | 1 | **Init** | CLI parsing, interactive folder selection (if no arg), config validation, logger setup, load learning insights, route to dynamic/progress mode |
 | 2 | **Discover** | Find videos/audio, discover documents, resolve user name, check resume state |
 | 3 | **Services** | Firebase auth, Gemini init, prepare document parts |
+| 3.5 | **Deep Summary** | (optional) Pre-summarize context docs with Gemini — 60-80% token savings |
 | 4 | **Process** | Compress → Upload → Analyze → Quality Gate → Retry → Focused Pass |
 | 5 | **Compile** | Cross-segment compilation, diff engine comparison |
 | 6 | **Output** | Write JSON, render Markdown + HTML, upload to Firebase |
@@ -199,7 +199,7 @@ Each video segment goes through this flow (Phase 4 detail):
 
 ```mermaid
 flowchart TB
-    START(["Segment N"]) --> COMPRESS["ffmpeg compress\nH.264 CRF 24, 1.5x speed"]
+    START(["Segment N"]) --> COMPRESS["ffmpeg compress\nH.264 CRF 24, 1.6x speed"]
     COMPRESS --> VERIFY["Verify segment integrity"]
     VERIFY --> UPLOAD_FB["Upload to Firebase Storage\n→ download URL"]
 
@@ -563,7 +563,7 @@ JSONL structured format includes phase spans with timing metrics for observabili
 | **ffmpeg** | System binary | H.264 video compression + segmentation |
 | **Git** | System binary | Change detection for progress tracking |
 
-**Codebase: ~45 files · ~13,000+ lines** · npm package: `task-summary-extractor` · CLI: `taskex`
+**Codebase: ~48 files · ~13,600+ lines** · npm package: `task-summary-extractor` · CLI: `taskex`
 
 ---
 
@@ -634,8 +634,8 @@ The project includes a comprehensive test suite using [vitest](https://vitest.de
 
 | Metric | Value |
 |--------|-------|
-| Test files | 13 |
-| Total tests | 285 |
+| Test files | 15 |
+| Total tests | 331 |
 | Framework | vitest v4.x |
 | Coverage | `@vitest/coverage-v8` |
 
@@ -662,45 +662,45 @@ npm run test:coverage # Coverage report
 |-----|-------------|
 | 📖 [README.md](README.md) | Setup, CLI flags, configuration, features |
 | 📖 [QUICK_START.md](QUICK_START.md) | Step-by-step first-time walkthrough |
-| 🔭 [EXPLORATION.md](EXPLORATION.md) | Module map, line counts, future roadmap |
 
 ---
 
-## JSON Schema Validation
-
-All AI output is validated against JSON Schema definitions in `src/schemas/`:
-
-| Schema | File | Purpose |
-|--------|------|---------|
-| Segment analysis | `analysis-segment.schema.json` | Validates each segment's extracted data |
-| Compiled analysis | `analysis-compiled.schema.json` | Validates the final cross-segment compilation |
+## Deep Summary
 
-Validation is performed by `src/utils/schema-validator.js` using [ajv](https://ajv.js.org/). Validation errors are reported as warnings with contextual hints for the retry/focused-pass cycle — they do not hard-fail the pipeline but are injected as corrective hints when the quality gate triggers a retry.
-
----
+The `--deep-summary` flag (or interactive prompt when many docs are detected) pre-summarizes context documents before segment analysis:
 
-## Test Suite
+```mermaid
+flowchart TB
+    START(["Context Docs"]) --> PARTITION["Partition: summarize vs. keep full"]
+    PARTITION --> SKIP["Skip tiny docs (<500 chars)"]
+    PARTITION --> EXCL["Excluded docs → keep full fidelity"]
+    PARTITION --> TO_SUM["Docs to summarize"]
+    TO_SUM --> TRUNC["Truncate oversized docs (>900K chars)"]
+    TRUNC --> BATCH["Group into batches\n(≤600K chars each)"]
+    BATCH --> AI["Gemini summarization\n(per batch)"]
+    AI --> REPLACE["Replace full content\nwith condensed summaries"]
+    REPLACE --> OUT(["Token-efficient\ncontext docs"])
+```
 
-The project includes a comprehensive test suite using [vitest](https://vitest.dev/):
+| Constant | Value | Purpose |
+|----------|-------|---------|
+| `BATCH_MAX_CHARS` | 600,000 | Max input chars per summarization batch |
+| `MAX_DOC_CHARS` | 900,000 | Hard cap per-document before truncation |
+| `SUMMARY_MAX_OUTPUT` | 16,384 | Max output tokens per summarization call |
+| `MIN_SUMMARIZE_LENGTH` | 500 | Docs below this skip summarization |
 
-| Metric | Value |
-|--------|-------|
-| Test files | 13 |
-| Total tests | 285 |
-| Framework | vitest v4.x |
-| Coverage | `@vitest/coverage-v8` |
+Typical savings: 60-80% reduction in per-segment context tokens. The user can exclude specific docs from summarization via `--exclude-docs` or the interactive picker.
 
-**Test categories:**
+---
 
-| Directory | What's Tested |
-|-----------|---------------|
-| `tests/utils/` | Utility modules: adaptive-budget, cli, confidence-filter, context-manager, diff-engine, format, json-parser, progress-bar, quality-gate, retry, schema-validator |
-| `tests/renderers/` | Renderer modules: html, markdown |
+## Context Window Safety
 
-**Commands:**
+Safeguards to prevent context window overflow:
 
-```bash
-npm test              # Run all tests
-npm run test:watch    # Watch mode
-npm run test:coverage # Coverage report
-```
+| Safeguard | Where | What It Does |
+|-----------|-------|-------------|
+| **P0/P1 hard cap** | `context-manager.js` | Critical docs can't exceed 2× the token budget |
+| **VTT fallback cap** | `context-manager.js` | Full VTT fallback capped at 500K chars |
+| **Doc truncation** | `deep-summary.js` | Oversized docs truncated to 900K chars before summarization |
+| **Compilation pre-flight** | `gemini.js` | Estimates tokens before compilation; trims middle segments if >80% of context |
+| **RESOURCE_EXHAUSTED recovery** | `gemini.js` | On quota/context errors: waits 30s, sheds docs, retries with reduced input |

package/QUICK_START.md

@@ -223,6 +223,7 @@ my-project/runs/{timestamp}/
 | **Force Gemini File API** | `taskex --no-storage-url "my-meeting"` |
 | **Preview without running** | `taskex --dry-run "my-meeting"` |
 | **Deep dive docs** | `taskex --deep-dive "my-meeting"` |
+| **Pre-summarize docs** | `taskex --deep-summary "my-meeting"` |
 | **Generate docs (no video)** | `taskex --dynamic "my-project"` |
 | **Track progress via git** | `taskex --update-progress --repo "C:\project" "my-meeting"` |
 | **Debug mode** | `taskex --log-level debug "my-meeting"` |
@@ -272,4 +273,3 @@ Your recordings, `.env`, logs — everything local is `.gitignore`d and safe.
 |------|-------|
 | Full feature list, all CLI flags, configuration | [README.md](README.md) |
 | How the pipeline works internally | [ARCHITECTURE.md](ARCHITECTURE.md) |
-| Module map, line counts, roadmap | [EXPLORATION.md](EXPLORATION.md) |

package/README.md

@@ -1,13 +1,13 @@
 # Task Summary Extractor
 
-> **v9.0.0** — AI-powered content analysis CLI — meetings, recordings, documents, or any mix. Install globally, run anywhere.
+> **v9.4.0** — AI-powered content analysis CLI — meetings, recordings, documents, or any mix. Install globally, run anywhere.
 
 <p align="center">
   <img src="https://img.shields.io/badge/node-%3E%3D18.0.0-green" alt="Node.js" />
   <img src="https://img.shields.io/badge/gemini-2.5--flash-blue" alt="Gemini" />
   <img src="https://img.shields.io/badge/firebase-11.x-orange" alt="Firebase" />
-  <img src="https://img.shields.io/badge/version-9.0.0-brightgreen" alt="Version" />
-  <img src="https://img.shields.io/badge/tests-285%20passing-brightgreen" alt="Tests" />
+  <img src="https://img.shields.io/badge/version-9.4.0-brightgreen" alt="Version" />
+  <img src="https://img.shields.io/badge/tests-331%20passing-brightgreen" alt="Tests" />
   <img src="https://img.shields.io/badge/npm-task--summary--extractor-red" alt="npm" />
 </p>
 
@@ -62,6 +62,20 @@ taskex --update-progress --repo "C:\my-project" "my-meeting"
 
 > **v7.2.3**: If the call folder isn't a git repo, the tool auto-initializes one for baseline tracking.
 
+### ⚡ Deep Summary (`--deep-summary`)
+
+Pre-summarize context documents to reduce per-segment token usage by 60-80%:
+
+```bash
+taskex --deep-summary --name "Jane" "my-meeting"
+```
+
+Exclude specific docs from summarization (keep at full fidelity):
+
+```bash
+taskex --deep-summary --exclude-docs "code-map.md,sprint.md" "my-meeting"
+```
+
 > See all modes explained with diagrams → [ARCHITECTURE.md](ARCHITECTURE.md#pipeline-phases)
 
 ---
@@ -172,6 +186,8 @@ These are the ones you'll actually use:
 | `--format <type>` | Output format: `md`, `html`, `json`, `pdf`, `docx`, `all` (default: `md`) | `--format html` |
 | `--min-confidence <level>` | Filter items by confidence: `high`, `medium`, `low` | `--min-confidence high` |
 | `--no-html` | Suppress HTML report generation | `--no-html` |
+| `--deep-summary` | Pre-summarize context docs (60-80% token savings) | `--deep-summary` |
+| `--exclude-docs <list>` | Docs to keep full during deep-summary (comma-separated) | `--exclude-docs "code-map.md"` |
 
 **Typical usage:**
 
@@ -198,6 +214,7 @@ Choose what the tool does. Only use one at a time:
 | *(none)* | **Content analysis** | `results.md` + `results.html` — structured task document |
 | `--dynamic` | **Doc generation** | `INDEX.md` + 3–15 topic documents |
 | `--deep-dive` | **Topic explainers** | `INDEX.md` + per-topic deep-dive docs |
+| `--deep-summary` | **Token-efficient analysis** | Same as content analysis, but context docs pre-summarized (60-80% savings) |
 | `--update-progress` | **Progress check** | `progress.md` — item status via git |
 
 **Dynamic mode** also uses:
@@ -259,7 +276,7 @@ taskex [flags] [folder]
 
 CONFIG     --gemini-key  --firebase-key  --firebase-project
            --firebase-bucket  --firebase-domain
-MODES      --dynamic  --deep-dive  --update-progress
+MODES      --dynamic  --deep-dive  --deep-summary  --update-progress
 CORE       --name  --model  --skip-upload  --resume  --reanalyze  --dry-run
 OUTPUT     --format <md|html|json|pdf|docx|all>  --min-confidence <high|medium|low>
            --no-html
@@ -394,7 +411,7 @@ GEMINI_API_KEY=your-key-here
 
 # Optional — uncomment to customize
 # GEMINI_MODEL=gemini-2.5-flash
-# VIDEO_SPEED=1.5
+# VIDEO_SPEED=1.6
 # THINKING_BUDGET=24576
 # LOG_LEVEL=info
 
@@ -413,7 +430,7 @@ GEMINI_API_KEY=your-key-here
 
 | Feature | Description |
 |---------|-------------|
-| **Video/Audio Compression** | H.264 CRF 24, text-optimized sharpening, configurable speed |
+| **Video/Audio Compression** | H.264 CRF 24, text-optimized sharpening, 1.6× speed |
 | **Smart Segmentation** | ≤5 min chunks with boundary-aware splitting |
 | **Cross-Segment Continuity** | Ticket IDs, names, and context carry forward |
 | **Document Discovery** | Auto-finds docs in all subfolders |
@@ -434,6 +451,8 @@ GEMINI_API_KEY=your-key-here
 | **HTML Report** | Self-contained HTML report with collapsible sections, filtering, dark mode |
 | **JSON Schema Validation** | Validates AI output against JSON Schema (segment + compiled) |
 | **Confidence Filter** | `--min-confidence` flag to exclude low-confidence items from output |
+| **Deep Summary** | `--deep-summary` pre-summarizes context docs, 60-80% token savings per segment |
+| **Context Window Safety** | Auto-truncation, pre-flight token checks, RESOURCE_EXHAUSTED recovery |
 | **Multi-Format Output** | `--format` flag: Markdown, HTML, JSON, PDF, DOCX, or all formats at once |
 | **Interactive CLI** | Run with no args → guided experience |
 | **Resume / Checkpoint** | `--resume` continues interrupted runs |
@@ -507,6 +526,7 @@ task-summary-extractor/
 │   │   ├── git.js              Git CLI wrapper
 │   │   └── doc-parser.js       Document text extraction (DOCX, XLSX, PPTX, etc.)
 │   ├── modes/                  AI-heavy pipeline phase modules
+│   │   ├── deep-summary.js     Pre-summarize context docs (deep-summary feature)
 │   │   ├── deep-dive.js        Topic discovery & deep-dive doc generation
 │   │   ├── dynamic-mode.js     Dynamic document planning & generation
 │   │   ├── focused-reanalysis.js  Targeted reanalysis of weak segments
@@ -528,17 +548,14 @@ task-summary-extractor/
 │       ├── schema-validator.js JSON Schema validation (ajv)
 │       └── ... (15 more utility modules)
 │
-├── tests/                      Test suite — 285 tests across 13 files (vitest)
+├── tests/                      Test suite — 331 tests across 15 files (vitest)
 │   ├── utils/                  Utility module tests
 │   └── renderers/              Renderer tests
 │
 ├── QUICK_START.md              Step-by-step setup guide
-├── ARCHITECTURE.md             Technical deep dive
-└── EXPLORATION.md              Roadmap & future features
+└── ARCHITECTURE.md             Technical deep dive
 ```
 
-> Full module map with line counts → [EXPLORATION.md](EXPLORATION.md#full-module-map)
-
 ---
 
 ## npm Scripts
@@ -551,7 +568,7 @@ task-summary-extractor/
 | `npm run check` | Validate environment |
 | `npm start` | Run the pipeline |
 | `npm run help` | Show CLI help |
-| `npm test` | Run test suite (285 tests) |
+| `npm test` | Run test suite (331 tests) |
 | `npm run test:watch` | Run tests in watch mode |
 | `npm run test:coverage` | Run tests with coverage report |
 
@@ -561,6 +578,9 @@ task-summary-extractor/
 
 | Version | Highlights |
 |---------|-----------|
+| **v9.4.0** | **Context window safety** — pre-flight token checks, auto-truncation for oversized docs/VTTs, RESOURCE_EXHAUSTED recovery with automatic doc shedding, chunked compilation for large segment sets, P0/P1 hard cap (2× budget) prevents context overflow, improved deep-summary prompt quality |
+| **v9.3.1** | **Audit & polish** — VIDEO_SPEED 1.5→1.6, `--exclude-docs` flag for non-interactive deep-summary exclusion, friendlier Gemini error messages, dead code removal, DRY RUN_PRESETS |
+| **v9.3.0** | **Deep summary** — `--deep-summary` pre-summarizes context documents (60-80% token savings), interactive doc picker, `--exclude-docs` for CLI automation, batch processing |
 | **v9.0.0** | **CLI UX upgrade** — colors & progress bar, HTML reports, PDF & DOCX output (via puppeteer and docx npm package), JSON Schema validation, confidence filter (`--min-confidence`), pipeline decomposition (`src/phases/` — 9 modules), test suite (285 tests via vitest), multi-format output (`--format`: md/html/json/pdf/docx/all), doc-parser service, shared renderer utilities |
 | **v8.3.0** | **Universal content analysis** — prompt v4.0.0 supports video, audio, documents, and mixed content; input type auto-detection; timestamps conditional on content type; gemini.js bridge text generalized; all markdown docs updated |
 | **v8.2.0** | **Architecture cleanup** — `src/modes/` for AI pipeline phases, `retry.js` self-contained defaults, dead code removal, export trimming, `process_and_upload.js` slim shim, `progress.js` → `checkpoint.js`, merged `prompt.js` into `cli.js` |
@@ -587,7 +607,6 @@ task-summary-extractor/
 |-----|-------------|-------------|
 | 📖 **[QUICK_START.md](QUICK_START.md)** | Full setup walkthrough, examples, troubleshooting | First time using the tool |
 | 🏗️ **[ARCHITECTURE.md](ARCHITECTURE.md)** | Pipeline phases, algorithms, Mermaid diagrams | Understanding how it works |
-| 🔭 **[EXPLORATION.md](EXPLORATION.md)** | Module map, line counts, future roadmap | Contributing or extending |
 
 ---
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "task-summary-extractor",
-  "version": "9.2.2",
+  "version": "9.4.0",
   "description": "AI-powered meeting analysis & document generation CLI — video + document processing, deep dive docs, dynamic mode, interactive CLI with model selection, confidence scoring, learning loop, git progress tracking",
   "main": "process_and_upload.js",
   "bin": {
@@ -15,8 +15,7 @@
     ".env.example",
     "README.md",
     "QUICK_START.md",
-    "ARCHITECTURE.md",
-    "EXPLORATION.md"
+    "ARCHITECTURE.md"
   ],
   "scripts": {
     "setup": "node setup.js",

package/src/config.js

@@ -220,7 +220,7 @@ function getMaxThinkingBudget() {
 
 // ======================== VIDEO PROCESSING ========================
 
-const SPEED = envFloat('VIDEO_SPEED', 1.5);
+const SPEED = envFloat('VIDEO_SPEED', 1.6);
 const SEG_TIME = envInt('VIDEO_SEGMENT_TIME', 280); // seconds — produces segments < 5 min
 const PRESET = env('VIDEO_PRESET', 'slow');
 const VIDEO_EXTS = ['.mp4', '.mkv', '.avi', '.mov', '.webm'];