task-summary-extractor 8.3.0 → 9.0.0

package/EXPLORATION.md

@@ -1,6 +1,6 @@
 # Task Summary Extractor — Where We Are & Where We Can Go
 
-> **Version 8.3.0** — February 2026  
+> **Version 9.0.0** — March 2026  
 > Module map, codebase stats, and future roadmap.  
 > For setup and CLI reference, see [README.md](README.md) · [Quick Start](QUICK_START.md)  
 > For architecture diagrams and algorithms, see [ARCHITECTURE.md](ARCHITECTURE.md)
@@ -11,15 +11,15 @@
 
 ### Architecture Overview
 
-```
+```text
 ┌─────────────────────────────────────────────────────────────────────┐
 │          taskex (bin/taskex.js) or process_and_upload.js            │
 │                          (Entry Points)                             │
 └─────────────────────────┬───────────────────────────────────────────┘
                           │
 ┌─────────────────────────▼───────────────────────────────────────────┐
-│                       pipeline.js (1,738 lines)                     │
-│                    8-Phase Orchestrator                              │
+│                       pipeline.js (~920 lines)                      │
+│              9-Phase Orchestrator + src/phases/                      │
 │                                                                     │
 │  Init ──────► Discover ──► Services ──► ProcessVideo ──► Compile    │
 │  │ learning                               │    │           │        │
@@ -27,8 +27,8 @@
 │  │ loaded                           │ For each    │   Engine       │
 │  │                                  │ segment:    │      │         │
 │  │                                  │ ┌─────────┐ │   Output      │
-│  │                                  │ │Compress │ │      │         │
-│  │                                  │ │Upload   │ │   Health      │
+│  │                                  │ │Compress │ │   (MD+HTML)    │
+│  │                                  │ │Upload   │ │      │         │
 │  │                                  │ │Analyze ◄──┼── Quality Gate │
 │  │                                  │ │ ↻Retry  │ │   + Confidence│
 │  │                                  │ │ 🔍Focus │ │   Scoring     │
@@ -36,43 +36,47 @@
 │  │                                  └─────────────┘   Summary     │
 │  │                                                       │         │
 │  └──────────────────── learning history saved ◄──────────┘         │
-└────┬──────────┬──────────┬──────────┬──────────┬───────────────────┘
-     │          │          │          │          │
-┌────▼───┐ ┌───▼──────┐ ┌▼────────┐ ┌▼─────┐ ┌▼──────────┐
-│Services│ │  Utils   │ │Renderers│ │Logger│ │  Config   │
-│        │ │          │ │         │ │      │ │           │
-│gemini  │ │quality   │ │markdown │ │JSONL │ │dotenv     │
-│firebase│ │-gate     │ │(879 ln) │ │struct│ │validation │
-│video   │ │focused   │ │+ conf   │ │spans │ │env helpers│
-│git     │ │-reanalysis│ │  badges│ │phases│ │model reg  │
-│        │ │learning  │ │+ diff   │ │metrics│ │           │
-│        │ │-loop     │ │ section │ │      │ │           │
-│        │ │diff      │ │         │ │      │ │           │
-│        │ │-engine   │ │         │ │      │ │           │
-│        │ │adapt-budg│ │         │ │      │ │           │
-│        │ │context   │ │         │ │      │ │           │
-│        │ │+12 more  │ │         │ │      │ │           │
-└────────┘ └──────────┘ └─────────┘ └──────┘ └───────────┘
+└────┬──────────┬──────────┬──────────┬──────┬─────────┬─────────────┘
+     │          │          │          │      │         │
+┌────▼───┐ ┌───▼──────┐ ┌▼────────┐ ┌▼─────┐ ┌▼─────┐┌▼─────────┐
+│Services│ │  Utils   │ │Renderers│ │Logger│ │Schema││  Config  │
+│        │ │          │ │         │ │      │ │      ││          │
+│gemini  │ │quality   │ │markdown │ │JSONL │ │seg   ││dotenv    │
+│firebase│ │-gate     │ │(801 ln) │ │struct│ │comp  ││validation│
+│video   │ │colors    │ │html.js  │ │spans │ │      ││env helper│
+│git     │ │progress  │ │(673 ln) │ │phases│ │      ││model reg │
+│doc-    │ │-bar      │ │pdf.js   │ │metric│ │      ││          │
+│parser  │ │confidence│ │docx.js  │ │      │ │      ││          │
+│        │ │-filter   │ │shared   │ │      │ │      ││          │
+│        │ │          │ │(212 ln) │ │      │ │      ││          │
+│        │ │schema-   │ │         │ │      │ │      ││          │
+│        │ │validator │ │         │ │      │ │      ││          │
+│        │ │+15 more  │ │         │ │      │ │      ││          │
+└────────┘ └──────────┘ └─────────┘ └──────┘ └──────┘└──────────┘
 ```
 
 ### Codebase Stats
 
 | Category | Files | Lines |
-|----------|-------|-------|
-| Pipeline orchestrator | 1 | 1,985 |
-| Services (Gemini, Firebase, Video, Git) | 4 | 1,330 |
-| Utilities (21 modules) | 21 | 4,890 |
-| Renderers | 1 | 969 |
-| Config + Logger | 2 | 578 |
-| Entry points (taskex + legacy) | 2 | 165 |
-| Setup script | 1 | 505 |
-| Prompt (JSON) | 1 | 308 |
-| **Total** | **33 files** | **~10,600 lines** |
+| ---------- | ------- | ------- |
+| Pipeline orchestrator | 1 | ~920 |
+| Pipeline phases (`src/phases/`) | 9 | ~1,800 |
+| Services (Gemini, Firebase, Video, Git, Doc-Parser) | 5 | ~1,650 |
+| Modes (AI pipeline phases) | 5 | 2,054 |
+| Utilities (19 modules) | 19 | ~4,100 |
+| Renderers (markdown, html, pdf, docx, shared) | 5 | ~2,100 |
+| Config + Logger | 2 | 597 |
+| Schemas (JSON) | 2 | ~400 |
+| Entry points (taskex + legacy) | 2 | 79 |
+| Setup script | 1 | 418 |
+| Prompt (JSON) | 1 | 333 |
+| Tests (vitest) | 13 | ~3,200 |
+| **Total** | **~63 files** | **~13,000+ lines** |
 
 ### Version History
 
 | Version | Theme | Key Additions |
-|---------|-------|---------------|
+| --------- | ------- | --------------- |
 | **v3** | Core Improvements | dotenv, logger, retry logic, CLI args, graceful shutdown, config validation, progress persistence, error handling, video fixes, parallel uploads |
 | **v4** | Architecture & Orchestration | Phase decomposition (7 phases), CostTracker, configurable thinking budget, poll timeouts, dead code cleanup, no `process.exit()`, CLI enhancements |
 | **v5** | Smart & Accurate | Quality Gate (4-dimension scoring), auto-retry with corrective hints, adaptive thinking budget, smart boundary detection, health dashboard, enhanced prompt engineering, compilation quality assessment |
@@ -84,49 +88,55 @@
 | **v7.2** | Model Selection | Interactive model selector, `--model` flag, 5-model registry with pricing, runtime model switching |
 | **v7.2.1** | Storage URL + Audit | Firebase Storage URLs as Gemini External URLs (skip File API upload), 3-strategy file resolution, URI reuse for retry/focused pass, Gemini file cleanup, confidence % fix, logger/firebase/git/version fixes |
 | **v7.2.2** | Upload Control | `--force-upload` to re-upload existing files, `--no-storage-url` to force Gemini File API, production-ready docs |
+| **v7.2.3** | Production Hardening | Cross-platform ffmpeg detection, shell injection fix (spawnSync), auto git init for `--update-progress`, `runs/` excluded from doc discovery, entry point docs updated |
+| **v8.0.0** | npm Package | Global CLI (`taskex`), `--gemini-key`/`--firebase-*` config flags, CWD-based path resolution, CWD-first `.env`, `bin/taskex.js` entry point, npm publish-ready `package.json` |
 | **v8.1.0** | Smart Global Config | Persistent `~/.taskexrc` config, `taskex config` subcommand, first-run API key prompting, 5-level config resolution, production audit (14 fixes), shared CLI flag injection, boolean flag parser fix |
 | **v8.2.0** | Architecture Cleanup | `src/modes/` for AI pipeline phases, `retry.js` self-contained defaults, dead code removal, export trimming, `process_and_upload.js` slim shim, `progress.js` → `checkpoint.js`, merged `prompt.js` into `cli.js` |
 | **v8.3.0** | Universal Content Analysis | prompt.json v4.0.0 — input type auto-detection (video/audio/document/mixed), timestamps conditional, domain-adaptive extraction for any content source, gemini.js bridge text generalized |
-| **v8.0.0** | npm Package | Global CLI (`taskex`), `--gemini-key`/`--firebase-*` config flags, CWD-based path resolution, CWD-first `.env`, `bin/taskex.js` entry point, npm publish-ready `package.json` |
-| **v7.2.3** | Production Hardening | Cross-platform ffmpeg detection, shell injection fix (spawnSync), auto git init for `--update-progress`, `runs/` excluded from doc discovery, entry point docs updated |
+| **v9.0.0** | CLI UX + Pipeline Decomposition | Colors & progress bar, HTML reports (`results.html`), PDF & DOCX output (`results.pdf`, `results.docx`), JSON Schema validation (`src/schemas/`), confidence filter (`--min-confidence`), pipeline decomposed into `src/phases/` (9 modules), test suite (285 tests via vitest), multi-format output (`--format`: md/html/json/pdf/docx/all), doc-parser service, shared renderer utilities |
 
 ### What v6 Delivers
 
 #### 1. Confidence Scoring Per Extracted Item
+
 Every ticket, action item, CR, blocker, and scope change now carries a `confidence` field (HIGH / MEDIUM / LOW) and a `confidence_reason` explaining the evidence basis.
 
 | Confidence | Meaning | Example |
-|------------|---------|---------|
+| ------------ | --------- | --------- |
 | **HIGH** | Explicitly stated + corroborated by docs/context | "Mentioned by name with ticket ID in VTT and Azure DevOps" |
 | **MEDIUM** | Partially stated or single-source | "Discussed verbally but no written reference" |
 | **LOW** | Inferred from context, not directly stated | "Implied from related discussion, not explicitly assigned" |
 
 **Where it shows up:**
-- **Quality Gate** (`quality-gate.js` — 430 lines): New 15-point confidence coverage dimension in density scoring. Flags missing confidence fields and suspicious uniformity (all HIGH = likely not calibrated). Generates retry hints for poor confidence.
-- **Markdown Renderer** (`markdown.js` — 969 lines): Confidence badges (🟢 🟡 🔴) on every ticket header, action item row, CR row, blocker, scope change, and todo item. "📊 Confidence Distribution" summary table near report header.
-- **Prompt** (`prompt.json` — 265 lines): Explicit confidence scoring instructions injected into extraction prompt. Self-verification checklist updated.
 
-#### 2. Focused Re-Analysis (`focused-reanalysis.js` — 318 lines)
+- **Quality Gate** (`quality-gate.js` — 366 lines): New 15-point confidence coverage dimension in density scoring. Flags missing confidence fields and suspicious uniformity (all HIGH = likely not calibrated). Generates retry hints for poor confidence.
+- **Markdown Renderer** (`markdown.js` — 879 lines): Confidence badges (🟢 🟡 🔴) on every ticket header, action item row, CR row, blocker, scope change, and todo item. "📊 Confidence Distribution" summary table near report header.
+- **Prompt** (`prompt.json` — 333 lines): Explicit confidence scoring instructions injected into extraction prompt. Self-verification checklist updated.
+
+#### 2. Focused Re-Analysis (`focused-reanalysis.js` — 268 lines)
+
 When the quality gate identifies specific weak dimensions (score <60, ≥2 weak areas), a **targeted second pass** runs instead of a full re-analysis.
 
 | Component | What It Does |
-|-----------|--------------|
+| ----------- | -------------- |
 | `identifyWeaknesses()` | Analyzes quality dimensions + confidence coverage to find gaps (missing tickets, sparse assignees, low confidence items, broken cross-refs) |
 | `runFocusedPass()` | Sends a focused Gemini prompt targeting ONLY the weak areas, with reduced thinking budget (12K tokens) |
 | `mergeFocusedResults()` | Intelligent merge: updates existing items by ID, appends new items, marks `_enhanced_by_focused_pass` / `_from_focused_pass` |
 
 **Pipeline integration**: Runs after the quality gate + retry cycle for each segment. Controlled by `--no-focused-pass` flag. Costs tracked separately in cost tracker.
 
-#### 3. Learning & Improvement Loop (`learning-loop.js` — 302 lines)
+#### 3. Learning & Improvement Loop (`learning-loop.js` — 269 lines)
+
 The pipeline remembers its past performance and auto-tunes for the future.
 
 **How it works:**
+
 1. **Before processing**: `loadHistory()` reads `history.json` (up to 50 past runs), `analyzeHistory()` computes trends and budget adjustments
 2. **Budget auto-tuning**: If avg quality <45 across recent runs → boost thinking budget +4096 tokens. If >80 → reduce by 2048 to save cost.
 3. **Retry effectiveness**: Tracks whether retries actually improve quality. If retry success rate <30%, recommends increasing base budget instead.
 4. **After processing**: `saveHistory()` persists compact metrics (quality scores, extraction counts, costs, budgets, retry stats) for the next run.
 
-```
+```text
   📈 Learning Insights:
     Historical runs : 12
     Quality trend   : improving (avg: 74/100)
@@ -136,11 +146,12 @@ The pipeline remembers its past performance and auto-tunes for the future.
       • Focused re-analysis was used in 3/10 runs — system is self-correcting effectively
 ```
 
-#### 4. Diff-Aware Compilation (`diff-engine.js` — 316 lines)
+#### 4. Diff-Aware Compilation (`diff-engine.js` — 277 lines)
+
 Compares the current run's compiled analysis against the previous run to produce a delta report.
 
 | Diff Category | What's Detected |
-|---------------|-----------------|
+| --------------- | ----------------- |
 | **New items** | Tickets, CRs, action items, blockers, scope changes that didn't exist before |
 | **Removed items** | Items from the previous run that no longer appear |
 | **Changed items** | Status, priority, assignee, or confidence changes on existing items |
@@ -148,23 +159,27 @@ Compares the current run's compiled analysis against the previous run to produce
 
 **Output**: Appended to `results.md` as a "🔄 Changes Since Previous Run" section with summary table + detailed new/removed/changed listings. Also saved as `diff.json` in the run folder.
 
-#### 5. Structured Logging & Observability (`logger.js` — 352 lines)
+#### 5. Structured Logging & Observability (`logger.js` — 306 lines)
+
 The logger now writes **three parallel outputs**:
 
 | Output | Format | Purpose |
-|--------|--------|---------|
+| -------- | -------- | --------- |
 | `*_detailed.log` | Human-readable | Full debug/info/warn/error messages |
 | `*_minimal.log` | Compact steps | Steps + timestamps only |
 | `*_structured.jsonl` | Machine-readable JSONL | Every event as a JSON object with level, timestamp, context, phase |
 
 **New capabilities:**
+
 - **Phase spans**: `phaseStart(name)` / `phaseEnd(meta)` track timing per pipeline phase with structured records
 - **Operation context**: `setContext()` / `clearContext()` stack for enriching log entries with segment/operation metadata
 - **Structured metrics**: `metric(name, value)` for recording quantitative data (confidence coverage, token counts, etc.)
 - All phase timers auto-emit structured span events
 
-#### 6. Enhanced Quality Gate (`quality-gate.js` — 430 lines)
+#### 6. Enhanced Quality Gate (`quality-gate.js` — 366 lines)
+
 **New in v6:** Confidence coverage is now a scoring dimension within density (15 points):
+
 - Checks percentage of items with valid confidence fields
 - Detects suspicious uniformity (all same confidence = likely not calibrated)
 - New `getConfidenceStats(analysis)` export returns `{total, high, medium, low, missing, coverage}`
@@ -173,7 +188,7 @@ The logger now writes **three parallel outputs**:
 ### All v5 Features Retained
 
 | Feature | Module | Description |
-|---------|--------|-------------|
+| --------- | -------- | ------------- |
 | Quality Gate | `quality-gate.js` | 4-dimension scoring (structure, density, integrity, cross-refs), auto-retry on FAIL |
 | Adaptive Thinking Budget | `adaptive-budget.js` | Segment position, complexity, context docs → dynamic 8K–32K range |
 | Smart Boundary Detection | `context-manager.js` | Mid-conversation detection, open ticket carry-forward, continuity hints |
@@ -183,7 +198,7 @@ The logger now writes **three parallel outputs**:
 ### Current Capabilities
 
 | Capability | Status | Description |
-|------------|--------|-------------|
+| ------------ | -------- | ------------- |
 | Video compression | ✅ Mature | ffmpeg-based, CRF, configurable speed/preset |
 | Video segmentation | ✅ Mature | Time-based splitting, segment pre-validation |
 | Firebase upload | ✅ Mature | Parallel, retry, skip-existing, anonymous auth, async I/O, `--force-upload` re-upload |
@@ -207,7 +222,7 @@ The logger now writes **three parallel outputs**:
 
 ### CLI Reference
 
-```
+```text
 Usage: taskex [options] [folder]
 
 Install: npm i -g task-summary-extractor
@@ -257,6 +272,11 @@ Tuning:
   --no-learning                     Disable learning loop
   --no-diff                         Disable diff comparison
 
+Output:
+  --format <type>                   Output format: md, html, json, pdf, docx, all (default: md)
+  --min-confidence <level>          Filter by confidence: high, medium, low
+  --no-html                         Suppress HTML report generation
+
 Info:
   --help, -h                        Show help
   --version, -v                     Show version
@@ -264,14 +284,24 @@ Info:
 
 ### Full Module Map
 
-```
+```text
 bin/
-└── taskex.js                 72 ln  ★ v8.0.0 — Global CLI entry point, config flag injection
+└── taskex.js                 65 ln  ★ v8.0.0 — Global CLI entry point, config flag injection
 
 src/
-├── config.js                277 ln  Central config, env vars, model registry, validation
+├── config.js                291 ln  Central config, env vars, model registry, validation
 ├── logger.js                306 ln  ★ v6 — Triple output: detailed + minimal + structured JSONL, phase spans, metrics
-├── pipeline.js            1,985 ln  Multi-mode orchestrator with Storage URL optimization, upload control flags, learning loop, focused re-analysis, diff engine, deep-dive, dynamic, auto git init
+├── pipeline.js            ~920 ln  Multi-mode orchestrator with lazy phase imports, Storage URL optimization, upload control flags, learning loop, diff engine
+├── phases/                         ★ v9.0.0 — Decomposed pipeline phase modules
+│   ├── _shared.js                  Shared phase utilities (logging, error helpers)
+│   ├── init.js                     Phase 1: CLI parsing, config validation, logger setup
+│   ├── discover.js                 Phase 2: Find videos/audio, discover docs, resolve name
+│   ├── services.js                 Phase 3: Firebase auth, Gemini init, doc prep
+│   ├── process-media.js            Phase 4: Compress, upload, analyze, quality gate
+│   ├── compile.js                  Phase 5: Cross-segment compilation, diff engine
+│   ├── output.js                   Phase 6: Write JSON, render MD + HTML
+│   ├── summary.js                  Phase 8: Save learning history, print summary
+│   └── deep-dive.js                Phase 9: Optional deep-dive generation
 ├── modes/                          ★ v8.2.0 — AI-heavy pipeline phase modules
 │   ├── change-detector.js   417 ln  Git-based change correlation engine
 │   ├── deep-dive.js         473 ln  ★ v6.2 — Topic discovery, parallel doc generation, index builder
@@ -279,32 +309,45 @@ src/
 │   ├── focused-reanalysis.js 268 ln ★ v6 — Weakness detection, targeted second pass, intelligent merge
 │   └── progress-updater.js  402 ln  ★ v6.1 — AI-powered progress assessment, status report generation
 ├── renderers/
-│   └── markdown.js          879 ln  ★ v6 — Confidence badges (🟢🟡🔴), confidence distribution table, diff section
+│   ├── markdown.js          801 ln  ★ v6 — Confidence badges (🟢🟡🔴), confidence distribution table, diff section
+│   ├── html.js              673 ln  ★ v9.0.0 — Self-contained HTML report: collapsible sections, confidence badges, filtering, dark mode
+│   ├── pdf.js                       ★ v9.0.0 — PDF report renderer: HTML → PDF conversion via puppeteer
+│   ├── docx.js                      ★ v9.0.0 — DOCX report renderer: programmatic Word document via docx npm package
+│   └── shared.js            212 ln  ★ v9.0.0 — Shared renderer utilities (name clustering, dedup, formatting)
 ├── services/
 │   ├── firebase.js           92 ln  Init, upload, exists check (with retry, async I/O)
 │   ├── gemini.js            677 ln  ★ v7.2.1 — 3-strategy file resolution, External URL support, cleanup, doc prep, analysis, compilation
-│   ├── git.js               310 ln  ★ v7.2.3 — Git CLI wrapper: log, diff, status, changed files, auto-init
-│   └── video.js             285 ln  ★ v7.2.3 — ffmpeg compress, segment, probe (cross-platform, spawnSync)
+│   ├── git.js               264 ln  ★ v7.2.3 — Git CLI wrapper: log, diff, status, changed files, auto-init
+│   ├── video.js             273 ln  ★ v7.2.3 — ffmpeg compress, segment, probe (cross-platform, spawnSync)
+│   └── doc-parser.js        346 ln  ★ v9.0.0 — Document text extraction (DOCX, XLSX, PPTX, HTML, ODT, RTF, EPUB)
 └── utils/                          Pure utilities — parsing, retry, budget, config
-    ├── adaptive-budget.js   232 ln  ★ v5 — Transcript complexity → dynamic budget
+    ├── adaptive-budget.js   230 ln  ★ v5 — Transcript complexity → dynamic budget
     ├── checkpoint.js        145 ln  Checkpoint/resume persistence (renamed from progress.js in v8.2.0)
-    ├── cli.js               415 ln  ★ v8.0.0 — Interactive prompts, model selector, folder picker, config flags, taskex help
-    ├── context-manager.js   424 ln  4-tier priority, VTT slicing, progressive context, boundary detection
+    ├── cli.js               391 ln  ★ v8.0.0 — Interactive prompts, model selector, folder picker, config flags, taskex help
+    ├── context-manager.js   420 ln  4-tier priority, VTT slicing, progressive context, boundary detection
     ├── cost-tracker.js      140 ln  Token counting, USD cost estimation (+ focused pass tracking)
-    ├── diff-engine.js       280 ln  ★ v6 — Cross-run delta: new/removed/changed items, Markdown rendering
+    ├── diff-engine.js       277 ln  ★ v6 — Cross-run delta: new/removed/changed items, Markdown rendering
     ├── format.js             27 ln  Duration, bytes formatting
-    ├── fs.js                 40 ln  Recursive doc finder (skips runs/)
-    ├── global-config.js     320 ln  ★ v8.1.0 — ~/.taskexrc persistent config, interactive setup
+    ├── fs.js                 34 ln  Recursive doc finder (skips runs/)
+    ├── global-config.js     274 ln  ★ v8.1.0 — ~/.taskexrc persistent config, interactive setup
     ├── health-dashboard.js  191 ln  ★ v5 — Quality report, density bars, efficiency metrics
-    ├── inject-cli-flags.js   58 ln  ★ v8.1.0 — CLI flag → env var injection
+    ├── inject-cli-flags.js   49 ln  ★ v8.1.0 — CLI flag → env var injection
     ├── json-parser.js       216 ln  5-strategy JSON extraction + repair
-    ├── learning-loop.js     270 ln  ★ v6 — History I/O, trend analysis, budget auto-tuning, recommendations
-    ├── quality-gate.js      372 ln  ★ v6 — 4+1 dimension scoring (+ confidence coverage), retry hints
-    └── retry.js             112 ln  Exponential backoff, parallel map (self-contained defaults)
-
-prompt.json                  308 ln  ★ v4.0.0 — Universal content analysis: video, audio, documents, mixed input; auto-detects input type + domain
+    ├── learning-loop.js     269 ln  ★ v6 — History I/O, trend analysis, budget auto-tuning, recommendations
+    ├── quality-gate.js      366 ln  ★ v6 — 4+1 dimension scoring (+ confidence coverage), retry hints
+    ├── retry.js             118 ln  Exponential backoff, parallel map (self-contained defaults)
+    ├── colors.js             84 ln  ★ v9.0.0 — Zero-dep ANSI color utility (bold, red, green, yellow, cyan, dim, reset)
+    ├── progress-bar.js      287 ln  ★ v9.0.0 — Visual progress bar with phase tracking, ETA, cost display, TTY-aware
+    ├── confidence-filter.js 130 ln  ★ v9.0.0 — Filter extracted items by confidence level (--min-confidence flag)
+    └── schema-validator.js  260 ln  ★ v9.0.0 — JSON Schema validation using ajv (segment + compiled schemas)
+
+schemas/
+├── analysis-segment.schema.json    ★ v9.0.0 — JSON Schema for segment analysis output
+└── analysis-compiled.schema.json   ★ v9.0.0 — JSON Schema for compiled analysis output
+
+prompt.json                  333 ln  ★ v4.0.0 — Universal content analysis: video, audio, documents, mixed input; auto-detects input type + domain
 process_and_upload.js         14 ln  Backward-compatible shim — delegates to bin/taskex.js
-setup.js                     505 ln  Automated first-time setup & environment validation (v8.0.0)
+setup.js                     418 ln  Automated first-time setup & environment validation (v8.0.0)
 ```
 
 ---
@@ -316,7 +359,7 @@ setup.js                     505 ln  Automated first-time setup & environment va
 The following features from the original exploration have been **fully implemented**:
 
 | Feature | Status | Implemented In |
-|---------|--------|----------------|
+| --------- | -------- | ---------------- |
 | 📊 Confidence Scoring Per Extracted Item | ✅ Done | `prompt.json`, `quality-gate.js`, `markdown.js` |
 | 🔄 Multi-Pass Analysis (Focused Re-extraction) | ✅ Done | `modes/focused-reanalysis.js` (268 ln), pipeline integration |
 | 🧠 Learning & Improvement Loop | ✅ Done | `learning-loop.js` (270 ln), pipeline init + save |
@@ -327,34 +370,44 @@ The following features from the original exploration have been **fully implement
 | 🗓️ Deep Dive Document Generation | ✅ Done | `modes/deep-dive.js` (473 ln), pipeline phase 9 |
 | 📝 Dynamic Mode (doc-only generation) | ✅ Done | `modes/dynamic-mode.js` (494 ln), pipeline `--dynamic` route |
 | 🤖 Runtime Model Selection | ✅ Done | `config.js` model registry, `cli.js` selector, `--model` flag |
+| 📊 Progress Bar | ✅ Done | `progress-bar.js` (287 ln), pipeline integration, TTY-aware |
+| 🌐 HTML Report Viewer | ✅ Done | `renderers/html.js` (673 ln), self-contained HTML with filtering, dark mode |
+| 🔧 JSON Schema Validation | ✅ Done | `schema-validator.js` (260 ln), `schemas/` (2 files), ajv-based |
+| 🎯 Confidence Filter | ✅ Done | `confidence-filter.js` (130 ln), `--min-confidence` flag |
+| 🏗️ Pipeline Decomposition | ✅ Done | `src/phases/` (9 modules), `pipeline.js` reduced to ~920 lines |
+| 🧪 Test Suite | ✅ Done | 285 tests across 13 files using vitest |
+| 🎨 ANSI Colors | ✅ Done | `colors.js` (84 ln), zero-dep ANSI color utility wired throughout CLI |
+| 📄 Doc Parser Service | ✅ Done | `doc-parser.js` (346 ln), DOCX/XLSX/PPTX/HTML/ODT/RTF/EPUB extraction |
 
 ---
 
 ### Tier 1: High-Impact, Medium Effort
 
 #### 🔊 Speaker Diarization & Attribution
+
 **What**: Automatically identify who is speaking at each moment in the video.  
 **How**: Use Gemini's audio understanding or integrate a dedicated diarization API (e.g., AssemblyAI, Deepgram) as a preprocessing step. Map speaker segments to VTT timestamps.  
 **Impact**: Dramatically improves action item attribution ("Mohamed said X" vs. "someone said X"). Currently relies on Gemini inferring speakers from VTT voice tags or contextual clues.  
 **Modules affected**: New `services/diarization.js`, updates to `gemini.js` content building, `context-manager.js` for speaker-aware slicing.
 
-#### 🌐 Web Dashboard / Viewer
-**What**: A browser-based UI to view analysis results interactively — filter by ticket, search, click timestamps to jump in video.  
-**How**: Generate a self-contained HTML file alongside the MD output (embed results.json). Or build a lightweight React/Next.js viewer that reads from Firebase.  
-**Impact**: Transforms the tool from CLI-output to a proper product. Stakeholders who don't use VS Code can access results.  
-**Modules affected**: New `renderers/html.js` or separate `viewer/` project. Firebase integration for hosted version.
+#### 🌐 ~~Web Dashboard / Viewer~~ ✅ Done (v9.0.0)
+
+**Status**: Implemented as `src/renderers/html.js` — self-contained HTML report with collapsible sections, confidence badges, filtering, dark mode, and print-friendly styling. Generated as `results.html` alongside `results.md`.
+**Next step**: Build a hosted React/Next.js viewer that reads from Firebase for team-wide access.
 
 ---
 
 ### Tier 2: Differentiation Features
 
 #### 🎯 Task Board Integration (Azure DevOps / Jira / Linear)
+
 **What**: Push extracted action items and tickets directly to your project management tool.  
 **How**: After compilation, map extracted items to work item templates. Use Azure DevOps REST API / Jira API / Linear API to create/update items. Cross-reference extracted CR numbers with existing work items.  
 **Impact**: Closes the loop — call discussions automatically become tracked work items. No manual "meeting notes → task creation" step.  
 **Modules affected**: New `services/task-board.js`, integration config in `config.js`, new CLI flags (`--push-to-jira`, `--sync-devops`).
 
 #### 🎙️ Real-Time / Live Analysis Mode
+
 **What**: Analyze calls as they happen, producing running analysis instead of post-call batch processing.  
 **How**: Stream audio/video chunks to Gemini in real-time using Live API. Maintain a rolling context window. Produce incremental analysis updates.  
 **Impact**: During the call, participants see extracted items appearing in real-time. Post-call report is instant.  
@@ -365,22 +418,26 @@ The following features from the original exploration have been **fully implement
 ### Tier 3: Platform Evolution
 
 #### 🏗️ Plugin Architecture
+
 **What**: Allow custom plugins for different output formats, analysis types, and integrations.  
 **How**: Define hook points: `onSegmentAnalyzed`, `onCompiled`, `onOutput`. Plugins register handlers. Ship with built-in plugins (markdown, json, firebase). Community can add: Slack notifications, email summaries, PDF reports, custom prompts per team.  
 **Impact**: Transforms from a single-purpose tool to a platform. Different teams customize for their workflow.
 
 #### 🤖 Multi-Model Support
+
 **What**: Support different AI models beyond Gemini — OpenAI GPT-4o, Claude, Llama local models.  
 **Status**: *Partially implemented in v7.2* — runtime model selection across 5 Gemini models with `--model` flag and interactive selector. Full multi-provider abstraction (OpenAI, Claude, local) remains a future enhancement.  
 **Next step**: Abstract the AI service behind a provider interface. Each provider implements: `upload()`, `analyze()`, `compile()`. Config selects the active provider.  
 **Impact**: Users choose the best model for their budget/accuracy needs. Can run local models for sensitive content. Enables A/B testing between models.
 
 #### 📱 Mobile App / Bot
+
 **What**: Telegram/Teams/Slack bot that accepts video links and returns analysis.  
 **How**: Bot receives a shared video/meeting link → triggers pipeline → sends back the compiled Markdown or a link to the web dashboard.  
 **Impact**: Zero-friction usage — share a link, get a task summary. No CLI needed.
 
 #### 🔐 Multi-Tenant SaaS
+
 **What**: Hosted version where teams sign up, configure their projects, and get analysis as a service.  
 **How**: Next.js frontend, Node.js API (reusing current pipeline), per-team Firebase/S3 storage, Stripe billing, queue-based processing.  
 **Impact**: Commercial product. Teams pay per call analyzed. Revenue model.
@@ -389,23 +446,25 @@ The following features from the original exploration have been **fully implement
 
 ### Tier 4: Polish & Reliability
 
-#### 🧪 Test Suite
-**What**: Unit tests for all utility modules, integration tests for the pipeline.  
-**How**: Jest or Vitest. Mock Gemini API responses. Test quality gate scoring with fixtures. Test JSON parser with known-bad inputs. Test adaptive budget calculations. Test focused-reanalysis merge logic. Test diff-engine comparisons. Test learning-loop trend analysis.  
-**Impact**: Confidence in changes. CI/CD pipeline with automated testing. Prevents regressions.  
-**Priority modules to test**: `quality-gate.js`, `adaptive-budget.js`, `json-parser.js`, `context-manager.js`, `cost-tracker.js`, `focused-reanalysis.js`, `diff-engine.js`, `learning-loop.js`.
+#### 🧪 ~~Test Suite~~ ✅ Done (v9.0.0)
+
+**Status**: 285 tests across 13 files using vitest. Covers: quality-gate, adaptive-budget, json-parser, confidence-filter, context-manager, diff-engine, format, progress-bar, retry, schema-validator, cli, and renderers (html, markdown).
+**Commands**: `npm test`, `npm run test:watch`, `npm run test:coverage`.
 
 #### 📦 ~~Packaging & Distribution~~ ✅ Done (v8.0.0)
+
 **Status**: Published as `task-summary-extractor` on npm. Global CLI: `taskex`. Install: `npm i -g task-summary-extractor`.  
 **What was done**: `bin/taskex.js` entry point, `--gemini-key`/`--firebase-*` CLI config flags, CWD-based `.env` resolution, `PKG_ROOT`/`PROJECT_ROOT` path split for global compatibility.
 
 #### 🔍 Advanced Observability (OpenTelemetry)
+
 **What**: Extend the existing structured JSONL logging with OpenTelemetry trace export for external monitoring.  
 **How**: Wrap existing `phaseStart`/`phaseEnd` spans with OTel SDK. Export traces to Jaeger/Grafana. Add alert rules on quality metric degradation.  
 **Impact**: Production monitoring. Performance profiling across runs. Alert on quality regression trends.  
 **Note**: Basic structured logging is already done in v6. This extends it to distributed tracing systems.
 
 #### 🌍 i18n Prompt Library
+
 **What**: Support different language pairs beyond Arabic+English. Ship prompt templates per domain.  
 **How**: Move prompt.json to a `prompts/` directory with variants: `arabic-english-dotnet.json`, `spanish-english-react.json`, `french-english-java.json`. CLI flag: `--prompt-template react-english`.  
 **Impact**: Anyone can use this tool regardless of their team's language or tech stack.
@@ -415,7 +474,7 @@ The following features from the original exploration have been **fully implement
 ### Quick Wins (< 1 day each)
 
 | Feature | Effort | Impact |
-|---------|--------|--------|
+| --------- | -------- | -------- |
 | **Email summary** — send compiled MD via SMTP after processing | ~2 hrs | Users get results in inbox |
 | **Slack webhook** — post summary to a channel | ~1 hr | Team-wide visibility |
 | **Segment preview** — show first 3 VTT lines per segment before analyzing | ~30 min | Better UX during processing |
@@ -423,7 +482,7 @@ The following features from the original exploration have been **fully implement
 | **~~Audio-only mode~~** | ~~Done~~ | prompt.json v4.0.0 supports audio/doc/mixed — pipeline video requirement is next |
 | **Watch mode** — monitor a folder and auto-process new recordings | ~3 hrs | Hands-free automation |
 | **Git integration** — auto-commit results to repo | ~1 hr | Version-controlled meeting history |
-| **Confidence threshold filter** — CLI flag to exclude LOW confidence items from output | ~1 hr | Cleaner reports on demand |
+| **Confidence threshold filter** | ~~Done~~ | ★ v9.0.0 `--min-confidence` flag implemented in `confidence-filter.js` |
 | **History viewer** — CLI command to print learning loop trends without running pipeline | ~2 hrs | Introspect past performance |
 
 ---
@@ -432,8 +491,8 @@ The following features from the original exploration have been **fully implement
 
 Based on impact vs. effort, here's a suggested 5-item sprint building on v7.2:
 
-1. **Test suite foundation** — Jest setup + tests for quality-gate, adaptive-budget, json-parser, focused-reanalysis, diff-engine, learning-loop (1.5 days)
-2. **Web dashboard / viewer** — Self-contained HTML report with filtering and video timestamp links (2 days)
+1. **~~Test suite foundation~~** — ✅ Done (v9.0.0) — 285 tests across 13 files using vitest
+2. **~~Web dashboard / viewer~~** — ✅ Done (v9.0.0) — Self-contained HTML report with filtering and collapsible sections
 3. **Speaker diarization** — Gemini audio understanding for speaker attribution (1.5 days)
 4. **Task board integration** — Push tickets/CRs to Azure DevOps or Jira (1.5 days)
 5. **Slack/email notification** — Post compiled results automatically (half day)
@@ -442,14 +501,14 @@ These five deliver: reliability (tests), accessibility (dashboard), accuracy (sp
 
 ---
 
-*Generated from the v8.0.0 codebase — 31 files, ~10,300 lines of self-improving pipeline intelligence. npm: `task-summary-extractor` · CLI: `taskex`*
+*Generated from the v9.0.0 codebase — ~63 files, ~13,000+ lines of self-improving pipeline intelligence. npm: `task-summary-extractor` · CLI: `taskex`*
 
 ---
 
 ## See Also
 
 | Doc | What's In It |
-|-----|-------------|
+| ----- | ------------- |
 | 📖 [README.md](README.md) | Setup, CLI flags, configuration, features |
 | 📖 [QUICK_START.md](QUICK_START.md) | Step-by-step first-time walkthrough |
 | 🏗️ [ARCHITECTURE.md](ARCHITECTURE.md) | Pipeline phases, processing flows, Mermaid diagrams |

package/QUICK_START.md

@@ -156,7 +156,9 @@ The pipeline will:
 4. **Analyze** each segment with Gemini AI — uses Firebase Storage URL directly when available (skips separate Gemini upload)
 5. **Quality check** — retry weak segments automatically (reuses file reference — no re-upload)
 6. **Compile** results across all segments
-7. **Output** `results.md` + `results.json`
+7. **Output** `results.md` + `results.html` + `results.json` (+ `results.pdf` / `results.docx` if requested via `--format`)
+
+> **Tip:** Use `--format html` to get only HTML output, `--format pdf` for PDF, `--format docx` for Word, or `--format all` for Markdown + HTML + JSON + PDF + DOCX. Use `--min-confidence high` to filter out low-confidence items.
 
 > **Tip:** Use `--force-upload` to re-upload files that already exist in Storage. Use `--no-storage-url` to bypass Storage URL optimization and force Gemini File API uploads.
 
@@ -169,6 +171,7 @@ This takes **~2-5 minutes** depending on video length.
 ```
 my-meeting/runs/{timestamp}/
 ├── results.md            ← Open this! Your task document
+├── results.html          ← Interactive HTML report (open in any browser)
 ├── results.json          ← Full pipeline data (JSON)
 └── compilation.json      ← All extracted items (JSON)
 ```
@@ -222,9 +225,9 @@ my-project/runs/{timestamp}/
 | **Deep dive docs** | `taskex --deep-dive "my-meeting"` |
 | **Generate docs (no video)** | `taskex --dynamic "my-project"` |
 | **Track progress via git** | `taskex --update-progress --repo "C:\project" "my-meeting"` |
+| **Debug mode** | `taskex --log-level debug "my-meeting"` |
 
 > **Tip:** If the call folder isn't already a git repo, `--update-progress` auto-initializes one for baseline tracking.
-| **Debug mode** | `taskex --log-level debug "my-meeting"` |
 
 > Full CLI reference with all flags → [README.md — CLI Flags](README.md#cli-flags)