task-summary-extractor 8.3.0 → 9.0.1

package/.env.example

@@ -0,0 +1,38 @@
+# ======================== FIREBASE ========================
+FIREBASE_API_KEY=your_firebase_api_key
+FIREBASE_AUTH_DOMAIN=your-project.firebaseapp.com
+FIREBASE_PROJECT_ID=your-project
+FIREBASE_STORAGE_BUCKET=your-project.appspot.com
+FIREBASE_MESSAGING_SENDER_ID=1234567890
+FIREBASE_APP_ID=1:1234567890:web:abc123
+FIREBASE_MEASUREMENT_ID=G-XXXXXXXXXX
+
+# ======================== GEMINI AI ========================
+GEMINI_API_KEY=your_gemini_api_key
+GEMINI_MODEL=gemini-2.5-flash
+
+# ======================== VIDEO PROCESSING ========================
+# Speed multiplier (default: 1.5)
+VIDEO_SPEED=1.5
+# Segment duration in seconds (default: 280)
+VIDEO_SEGMENT_TIME=280
+# ffmpeg preset: ultrafast, superfast, veryfast, faster, fast, medium, slow, slower, veryslow
+VIDEO_PRESET=slow
+
+# ======================== PIPELINE ========================
+# Log level: debug, info, warn, error (default: info)
+LOG_LEVEL=info
+# Max concurrent uploads (default: 3)
+MAX_PARALLEL_UPLOADS=3
+# Max retries for API calls (default: 3)
+MAX_RETRIES=3
+# Retry base delay in ms (default: 2000)
+RETRY_BASE_DELAY_MS=2000
+
+# ======================== GEMINI TUNING ========================
+# Thinking token budget per segment analysis (default: 24576)
+THINKING_BUDGET=24576
+# Thinking token budget for final compilation (default: 10240)
+COMPILATION_THINKING_BUDGET=10240
+# Max polling time for Gemini File API processing in ms (default: 300000 = 5 min)
+GEMINI_POLL_TIMEOUT_MS=300000

package/ARCHITECTURE.md

@@ -74,6 +74,7 @@ flowchart TB
         FB["firebase.js"]
         VID["video.js"]
         GIT["git.js"]
+        DP["doc-parser.js"]
     end
 
     subgraph Utils["Utilities"]
@@ -97,6 +98,10 @@ flowchart TB
 
     subgraph Renderers["Renderers"]
         MD["markdown.js"]
+        HTML["html.js"]
+        PDF["pdf.js"]
+        DOCX["docx.js"]
+        SHARED["shared.js"]
     end
 
     EP --> Pipeline
@@ -123,7 +128,7 @@ flowchart TB
 | 3 | **Services** | Firebase auth, Gemini init, prepare document parts |
 | 4 | **Process** | Compress → Upload → Analyze → Quality Gate → Retry → Focused Pass |
 | 5 | **Compile** | Cross-segment compilation, diff engine comparison |
-| 6 | **Output** | Write JSON, render Markdown, upload to Firebase |
+| 6 | **Output** | Write JSON, render Markdown + HTML, upload to Firebase |
 | 7 | **Health** | Quality metrics dashboard, cost breakdown |
 | 8 | **Summary** | Save learning history, print run summary |
 | 9 | **Deep Dive** | (optional, `--deep-dive`) Topic discovery + explanatory document generation |
@@ -170,6 +175,7 @@ flowchart LR
     subgraph P6["Phase 6: Output"]
         JSON["results.json"]
         MDR["results.md"]
+        HTMLR["results.html"]
         FBU["Firebase upload"]
     end
 
@@ -505,7 +511,10 @@ taskex --dynamic --request "Document this microservices architecture"
 | `.vtt` `.srt` `.txt` `.md` `.csv` | Inline text | Read and passed directly as text parts |
 | `.pdf` | Gemini File API | Uploaded as binary, Gemini processes natively |
 | `.mp3` `.wav` `.ogg` `.m4a` | Gemini File API | Uploaded as audio, Gemini processes natively |
-| `.docx` `.doc` | Firebase only | Uploaded for archival, not processable by Gemini |
+| `.docx` | Doc parser (mammoth) | Converted to plain text, sent as inline text |
+| `.xlsx` `.xls` | Doc parser (xlsx) | Converted to pipe-delimited tables, sent as inline text |
+| `.doc` `.pptx` `.ppt` `.odt` `.odp` `.ods` `.rtf` `.epub` | Doc parser (officeparser) | Converted to plain text, sent as inline text |
+| `.html` `.htm` | Doc parser (built-in) | HTML tags stripped, sent as inline text |
 
 Directories skipped during recursive discovery: `node_modules`, `.git`, `compressed`, `logs`, `gemini_runs`, `runs`
 
@@ -546,10 +555,15 @@ JSONL structured format includes phase spans with timing metrics for observabili
 | **Gemini AI** | `@google/genai@^1.42.0` | Video analysis, File API, 1M context window |
 | **Firebase** | `firebase@^12.9.0` | Anonymous auth + Cloud Storage uploads |
 | **dotenv** | `dotenv@^17.3.1` | Environment variable loading |
+| **puppeteer** | `puppeteer` | HTML → PDF conversion for PDF output format |
+| **docx** | `docx` | Programmatic Word document generation for DOCX output format |
+| **mammoth** | `mammoth` | DOCX → plain text conversion |
+| **xlsx** | `xlsx` | Excel spreadsheet parsing (XLSX/XLS) |
+| **officeparser** | `officeparser` | DOC, PPTX, ODT, RTF, EPUB text extraction |
 | **ffmpeg** | System binary | H.264 video compression + segmentation |
 | **Git** | System binary | Change detection for progress tracking |
 
-**Codebase: 31 files · ~10,300 lines** · npm package: `task-summary-extractor` · CLI: `taskex`
+**Codebase: ~45 files · ~13,000+ lines** · npm package: `task-summary-extractor` · CLI: `taskex`
 
 ---
 
@@ -601,6 +615,47 @@ When `usedExternalUrl` is `true`, the `fileUri` contains the Firebase Storage do
 
 ---
 
+## JSON Schema Validation
+
+All AI output is validated against JSON Schema definitions in `src/schemas/`:
+
+| Schema | File | Purpose |
+|--------|------|---------|
+| Segment analysis | `analysis-segment.schema.json` | Validates each segment's extracted data |
+| Compiled analysis | `analysis-compiled.schema.json` | Validates the final cross-segment compilation |
+
+Validation is performed by `src/utils/schema-validator.js` using [ajv](https://ajv.js.org/). Validation errors are reported as warnings with contextual hints for the retry/focused-pass cycle — they do not hard-fail the pipeline but are injected as corrective hints when the quality gate triggers a retry.
+
+---
+
+## Test Suite
+
+The project includes a comprehensive test suite using [vitest](https://vitest.dev/):
+
+| Metric | Value |
+|--------|-------|
+| Test files | 13 |
+| Total tests | 285 |
+| Framework | vitest v4.x |
+| Coverage | `@vitest/coverage-v8` |
+
+**Test categories:**
+
+| Directory | What's Tested |
+|-----------|---------------|
+| `tests/utils/` | Utility modules: adaptive-budget, cli, confidence-filter, context-manager, diff-engine, format, json-parser, progress-bar, quality-gate, retry, schema-validator |
+| `tests/renderers/` | Renderer modules: html, markdown |
+
+**Commands:**
+
+```bash
+npm test              # Run all tests
+npm run test:watch    # Watch mode
+npm run test:coverage # Coverage report
+```
+
+---
+
 ## See Also
 
 | Doc | What's In It |
@@ -608,3 +663,44 @@ When `usedExternalUrl` is `true`, the `fileUri` contains the Firebase Storage do
 | 📖 [README.md](README.md) | Setup, CLI flags, configuration, features |
 | 📖 [QUICK_START.md](QUICK_START.md) | Step-by-step first-time walkthrough |
 | 🔭 [EXPLORATION.md](EXPLORATION.md) | Module map, line counts, future roadmap |
+
+---
+
+## JSON Schema Validation
+
+All AI output is validated against JSON Schema definitions in `src/schemas/`:
+
+| Schema | File | Purpose |
+|--------|------|---------|
+| Segment analysis | `analysis-segment.schema.json` | Validates each segment's extracted data |
+| Compiled analysis | `analysis-compiled.schema.json` | Validates the final cross-segment compilation |
+
+Validation is performed by `src/utils/schema-validator.js` using [ajv](https://ajv.js.org/). Validation errors are reported as warnings with contextual hints for the retry/focused-pass cycle — they do not hard-fail the pipeline but are injected as corrective hints when the quality gate triggers a retry.
+
+---
+
+## Test Suite
+
+The project includes a comprehensive test suite using [vitest](https://vitest.dev/):
+
+| Metric | Value |
+|--------|-------|
+| Test files | 13 |
+| Total tests | 285 |
+| Framework | vitest v4.x |
+| Coverage | `@vitest/coverage-v8` |
+
+**Test categories:**
+
+| Directory | What's Tested |
+|-----------|---------------|
+| `tests/utils/` | Utility modules: adaptive-budget, cli, confidence-filter, context-manager, diff-engine, format, json-parser, progress-bar, quality-gate, retry, schema-validator |
+| `tests/renderers/` | Renderer modules: html, markdown |
+
+**Commands:**
+
+```bash
+npm test              # Run all tests
+npm run test:watch    # Watch mode
+npm run test:coverage # Coverage report
+```