@nathanvale/chatline 0.0.1

package/README.md

@@ -0,0 +1,1535 @@
+# Chatline
+
+> Extract, enrich, and render your iMessage conversations into beautiful,
+> AI-powered markdown timelines with full conversation threading and deep media
+> analysis.
+
+[![TypeScript](https://img.shields.io/badge/TypeScript-5.9+-3178c6?logo=typescript)](https://www.typescriptlang.org/)
+[![Node.js](https://img.shields.io/badge/Node.js-22%2B-339933?logo=nodedotjs)](https://nodejs.org/)
+[![License](https://img.shields.io/badge/License-MIT-blue)](#license)
+[![CI](https://github.com/nathanvale/chatline/actions/workflows/pr-quality.yml/badge.svg?branch=main)](https://github.com/nathanvale/chatline/actions/workflows/pr-quality.yml)
+[![CodeQL](https://github.com/nathanvale/chatline/actions/workflows/codeql.yml/badge.svg?branch=main)](https://github.com/nathanvale/chatline/actions/workflows/codeql.yml)
+[![Tests](https://img.shields.io/badge/Tests-50%2B-brightgreen)](#testing)
+[![Coverage](https://img.shields.io/badge/Coverage-70%25%2B-brightgreen)](#testing)
+
+## Overview
+
+**iMessage Timeline** is a sophisticated data pipeline that transforms your
+iMessage conversations into searchable, enriched markdown timelines. It
+intelligently extracts messages from multiple sources (iMazing CSV exports,
+macOS Messages.app SQLite database), deduplicates and links replies/reactions,
+enriches with AI-powered analysis (image descriptions, audio transcription, link
+summaries), and generates deterministic markdown files organized by date and
+time-of-day.
+
+Perfect for creating browsable conversation archives, enriched research notes,
+or personal history exports.
+
+## 📚 Documentation
+
+**Full documentation available at:**
+[https://nathanvale.github.io/chatline/](https://nathanvale.github.io/chatline/)
+
+The documentation site includes:
+
+- Getting Started Guide
+- CLI Usage
+- Pipeline Technical Specifications
+- Best Practices
+- Release Management
+- Troubleshooting
+
+### Automated Quality & Security
+
+This repository is continuously checked by:
+
+- **PR quality** workflow: lint, typecheck, full Vitest suite + V8 coverage,
+  delta quality checks
+- **CodeQL**: static analysis for code vulnerabilities (JS/TS)
+- **OSV Scanner**: open source vulnerability scanning of dependencies
+- **Workflow Lint**: actionlint validation of GitHub Actions syntax
+- **Dependency Review**: flags risky transitive additions on PRs
+- **Package Hygiene**: build integrity (`publint`, type checks, dry pack)
+- **Renovate**: automated npm/pnpm dependency maintenance (grouped, safe
+  automerge for dev minor/patch)
+- **Dependabot**: weekly GitHub Actions version bumps only
+
+Badges surface current status; failing checks block merges ensuring a
+high-signal, low-noise pipeline.
+
+### Releases and automation
+
+- Zero‑touch, canonical Changesets flow: CI opens a "Version Packages" PR,
+  auto‑merges it after required checks, publishes to npm (with tags), and
+  creates GitHub Releases with SBOM.
+- Pre‑releases: enter/exit beta/rc via a workflow; CI publishes -beta.N/-rc.N to
+  the matching npm dist‑tag.
+- Nightly snapshots (alpha): gated to run only while pre‑mode is active to avoid
+  confusion with stable releases.
+
+#### Publishing and NPM_TOKEN
+
+This repo uses Changesets to open a "Version Packages" PR and publish on merge.
+
+- If an npm token is configured (repository secret `NPM_TOKEN`):
+  - The Changesets workflow authenticates to npm and runs `pnpm release`
+  - A GitHub Release is created for the tagged version
+- If no npm token is configured:
+  - The workflow does NOT fail main
+  - It surfaces a clear warning in the logs and Job Summary:
+    - "NPM_TOKEN not set; skipping publish. Configure repository secret
+      'NPM_TOKEN' to enable publishing."
+  - Versioning PR behavior still works; only publish is skipped
+
+How to enable publishing:
+
+1. Create an npm access token with publish rights
+2. In GitHub → Settings → Secrets and variables → Actions → New repository
+   secret
+3. Name: `NPM_TOKEN`, Value: your token
+4. Re-run the "Changesets Manage & Publish" workflow on `main`
+
+Tip: You can trigger the workflow manually from the Actions tab or by:
+
+```bash
+gh workflow run "Changesets Manage & Publish" -r main
+```
+
+Note: The workflow also posts an annotation and adds a short section to the Job
+Summary whenever publish is skipped so it's visible in the GitHub UI without
+failing the pipeline.
+
+Docs:
+
+- Canonical flow: `docs/releases/changesets-canonical.md`
+- Branch protection policy: `docs/branch-protection-policy.md`
+- CI standards: `docs/ci-workflow-standards.md`
+
+### CI performance (optional)
+
+- setup-node already caches pnpm. If installs become a bottleneck, consider
+  adding a pnpm store cache step to PR quality (restore/save `.pnpm-store`). We
+  can wire this later once baseline run times are known.
+
+## CLI
+
+The project publishes a CLI executable `chatline` and provides a fast
+Bun-powered development loop.
+
+- Dev (TypeScript direct): `pnpm dev -- --help`
+- Built dist run: `pnpm cli -- --help`
+- Installed (after publish): `chatline --help`
+
+Docs:
+
+- Detailed usage: `docs/cli-usage.md`
+- Bun script rationale: `docs/bun-script-best-practices.md`
+
+Pass arguments after `--` when using `pnpm dev` or `pnpm cli`.
+
+### Key Features
+
+- **Multiple Sources**: Ingest from iMazing CSV exports and macOS Messages.app
+  database
+- **Intelligent Linking**: Automatically link replies to parents and associate
+  emoji reactions (tapbacks)
+- **Smart Deduplication**: Merge CSV/DB sources with GUID matching and content
+  equivalence detection
+- **AI Enrichment**:
+  - Image analysis (HEIC/TIFF→JPG previews + Gemini Vision captions)
+  - Audio transcription (with speaker labels and timestamps)
+  - PDF summarization
+  - Link context extraction (Firecrawl + provider fallbacks)
+- **Resumable Processing**: Checkpoint support for crash recovery and
+  incremental enrichment for processing only new messages
+- **Deterministic Output**: Identical input always produces identical markdown
+  (reproducible pipelines)
+- **Privacy-First**: Local-only mode, no API key persistence, full data control
+- **Conversation Threading**: Nested replies and tapbacks rendered as readable
+  blockquotes
+- **Type-Safe**: 100% TypeScript with Zod schema validation
+
+## For End Users
+
+### Installation & Setup
+
+#### Prerequisites
+
+- **Node.js** 22.20+
+- **macOS** (for database export; CSV import works on any OS)
+- **Gemini API Key** (for AI enrichment, get free at
+  https://aistudio.google.com)
+- **Firecrawl API Key** (optional, for link enrichment, get at
+  https://www.firecrawl.dev)
+
+#### Install Global CLI
+
+```bash
+npm install -g /chatline
+```
+
+This installs the `chatline` command globally, available from any
+directory.
+
+#### Environment Setup
+
+Create a `.env` file in your working directory:
+
+```bash
+GEMINI_API_KEY=your-api-key-here
+FIRECRAWL_API_KEY=your-api-key-here
+```
+
+Or export them in your shell:
+
+```bash
+export GEMINI_API_KEY=your-api-key-here
+export FIRECRAWL_API_KEY=your-api-key-here
+```
+
+#### Verify Installation
+
+```bash
+chatline doctor
+```
+
+Should show all checks passing.
+
+### Quick Start (Consumer)
+
+```bash
+# Initialize config (creates imessage-config.yaml)
+chatline init
+
+# Ingest CSV export from iMazing
+chatline ingest-csv -i messages.csv -o messages.csv.ingested.json
+
+# Ingest from macOS Messages.app database
+chatline ingest-db -i db-export.json -o messages.db.ingested.json
+
+# Normalize and link messages (merge sources, deduplicate, link replies)
+chatline normalize-link \
+  -i messages.csv.ingested.json messages.db.ingested.json \
+  -o messages.normalized.json
+
+# Enrich with AI (images, audio, links)
+chatline enrich-ai \
+  -i messages.normalized.json \
+  -o messages.enriched.json \
+  --enable-vision --enable-audio --enable-links
+
+# Render to markdown
+chatline render-markdown \
+  -i messages.enriched.json \
+  -o ./timeline
+```
+
+Output: A `timeline/` directory with daily markdown files, one per date.
+
+## Library Usage (Programmatic API)
+
+### Installation as Dependency
+
+Install as a library in your Node.js/TypeScript project:
+
+```bash
+npm install /chatline
+# or
+pnpm add /chatline
+# or
+yarn add /chatline
+```
+
+### TypeScript/JavaScript Import
+
+```typescript
+import {
+  // Config Management
+  loadConfig,
+  generateConfigContent,
+  validateConfig,
+
+  // Ingest Functions
+  ingestCSV,
+  dedupAndMerge,
+
+  // Utilities
+  detectDelta,
+  mergeEnrichments,
+
+  // Rate Limiting
+  createRateLimiter,
+
+  // Types
+  type Message,
+  type Config,
+  type DeltaResult,
+} from '@nathanvale/chatline'
+```
+
+### Example: Load and Validate Config
+
+```typescript
+import { loadConfig, validateConfig } from '@nathanvale/chatline'
+
+// Load config with auto-discovery (looks for imessage-config.yaml/json)
+const config = await loadConfig()
+
+// Load specific config file
+const config = await loadConfig({ configPath: './custom-config.yaml' })
+
+// Validate existing config object
+const validated = validateConfig({
+  gemini: { apiKey: 'your-key' },
+  inputs: { csv: ['messages.csv'] },
+})
+```
+
+### Example: Ingest Messages from CSV
+
+```typescript
+import { ingestCSV, createExportEnvelope } from '@nathanvale/chatline'
+import type { Message, IngestOptions } from '@nathanvale/chatline'
+
+const options: IngestOptions = {
+  attachmentDir: '/path/to/attachments',
+  strictMode: false,
+}
+
+const messages: Message[] = ingestCSV('./messages.csv', options)
+
+// Wrap in export envelope
+const envelope = createExportEnvelope(messages)
+console.log(`Ingested ${envelope.totalMessages} messages`)
+```
+
+### Example: Deduplicate and Merge Sources
+
+```typescript
+import { dedupAndMerge } from '@nathanvale/chatline'
+import type { Message } from '@nathanvale/chatline'
+
+const csvMessages: Message[] = ingestCSV('./messages.csv', options)
+const dbMessages: Message[] = JSON.parse(
+  fs.readFileSync('./db-export.json', 'utf-8'),
+)
+
+const result = dedupAndMerge(csvMessages, dbMessages)
+
+console.log(`Merged ${result.mergedCount} messages`)
+console.log(`Found ${result.stats.exactMatches} exact matches`)
+console.log(`Deduped ${result.stats.duplicatesRemoved} duplicates`)
+```
+
+### Example: Detect New Messages (Incremental Processing)
+
+```typescript
+import { detectDelta, extractGuidsFromMessages } from '@nathanvale/chatline'
+import type { Message, DeltaResult } from '@nathanvale/chatline'
+
+const currentMessages: Message[] = loadCurrentMessages()
+const previousMessages: Message[] = loadPreviousCheckpoint()
+
+const delta: DeltaResult = detectDelta(currentMessages, previousMessages)
+
+console.log(`New messages: ${delta.new.length}`)
+console.log(`Modified messages: ${delta.changed.length}`)
+console.log(`Removed messages: ${delta.removed.length}`)
+
+// Process only new messages
+const newGuids = extractGuidsFromMessages(delta.new)
+await enrichOnlyNew(newGuids)
+```
+
+### Example: Rate Limiting for API Calls
+
+```typescript
+import { createRateLimiter } from '@nathanvale/chatline'
+import type { RateLimitConfig } from '@nathanvale/chatline'
+
+const limiter = createRateLimiter({
+  requestsPerSecond: 10,
+  maxRetries: 3,
+  retryDelayMs: 1000,
+})
+
+// Use with fetch or any async API call
+const response = await limiter.execute(async () => {
+  return fetch('https://api.example.com/data')
+})
+
+console.log(`Status: ${response.status}`)
+```
+
+### Example: Generate Config Programmatically
+
+```typescript
+import { generateConfigContent, getDefaultConfigPath } from '@nathanvale/chatline'
+import fs from 'node:fs/promises'
+
+// Generate YAML config with defaults
+const yamlContent = generateConfigContent('yaml')
+const configPath = getDefaultConfigPath('yaml')
+
+await fs.writeFile(configPath, yamlContent, 'utf-8')
+console.log(`Config written to ${configPath}`)
+
+// Generate JSON config
+const jsonContent = generateConfigContent('json')
+await fs.writeFile('./imessage-config.json', jsonContent, 'utf-8')
+```
+
+### TypeScript Type Definitions
+
+The package includes full TypeScript definitions for all exports:
+
+```typescript
+import type {
+  // Core message types
+  Message,
+  MessageCore,
+  MediaMeta,
+  MediaEnrichment,
+  ReplyInfo,
+  TapbackInfo,
+
+  // Config types
+  Config,
+  ConfigFormat,
+
+  // Utility types
+  DeltaResult,
+  MergeStats,
+  IngestMergeResult,
+  EnrichmentMergeResult,
+
+  // Rate limiting types
+  RateLimitConfig,
+  RateLimitState,
+  ApiResponse,
+} from '@nathanvale/chatline'
+```
+
+### Advanced: Custom Pipeline
+
+```typescript
+import {
+  loadConfig,
+  ingestCSV,
+  dedupAndMerge,
+  detectDelta,
+  mergeEnrichments,
+  createRateLimiter,
+} from '@nathanvale/chatline'
+import type { Message, Config } from '@nathanvale/chatline'
+
+async function runCustomPipeline() {
+  // 1. Load configuration
+  const config: Config = await loadConfig()
+
+  // 2. Ingest from multiple sources
+  const csvMessages = ingestCSV(config.inputs.csv[0], {
+    attachmentDir: config.paths?.attachmentRoot,
+    strictMode: false,
+  })
+
+  const dbMessages = JSON.parse(await fs.readFile(config.inputs.db, 'utf-8'))
+
+  // 3. Merge and deduplicate
+  const merged = dedupAndMerge(csvMessages, dbMessages)
+  console.log(`Merged to ${merged.mergedCount} unique messages`)
+
+  // 4. Detect changes since last run
+  const previous = await loadPreviousState()
+  const delta = detectDelta(merged.messages, previous)
+
+  // 5. Enrich only new messages with rate limiting
+  const limiter = createRateLimiter({ requestsPerSecond: 5 })
+
+  for (const message of delta.new) {
+    if (message.media?.kind === 'image') {
+      const enrichment = await limiter.execute(() =>
+        enrichImageWithGemini(message),
+      )
+      message.media.enrichment = enrichment
+    }
+  }
+
+  // 6. Merge enrichments back into full dataset
+  const enriched = mergeEnrichments(merged.messages, delta.new)
+
+  // 7. Save checkpoint
+  await saveState(enriched.messages)
+
+  return enriched
+}
+```
+
+### See Also
+
+- **CLI Usage**: See `docs/cli-usage.md` for command-line interface examples
+- **Dual Distribution**: See `docs/dual-mode-distribution-best-practices.md` for
+  packaging details
+- **API Documentation**: See generated TypeDoc output (coming soon)
+
+## For Developers
+
+### Development Setup
+
+#### Clone & Install
+
+```bash
+# Clone the repository
+git clone https://github.com/yourusername/chatline.git
+cd chatline
+
+# Install dependencies
+pnpm install
+
+# Build TypeScript
+pnpm build
+```
+
+#### Local CLI Development
+
+During development, run the CLI directly from TypeScript via Bun, or run from
+the built `dist` output:
+
+```bash
+# Fast dev (TypeScript direct via Bun)
+pnpm dev -- --help
+pnpm dev -- --config examples/imessage-config.yaml
+
+# Or build, then run individual commands from dist
+pnpm build
+pnpm cli doctor
+pnpm cli ingest-csv -i messages.csv -o output.json
+
+# Watch mode for development (typecheck/build)
+pnpm watch
+```
+
+#### Running Tests
+
+```bash
+# Run all tests
+pnpm test
+
+# Watch mode
+pnpm test:watch
+
+# With UI
+pnpm test:ui
+
+# Coverage report
+pnpm coverage
+```
+
+#### Code Quality
+
+```bash
+# Lint code
+pnpm lint
+pnpm lint:fix
+
+# Format code
+pnpm format
+
+# Run quality checks (pre-commit hook)
+pnpm quality-check
+```
+
+### Bun-powered dev and tooling
+
+This project keeps pnpm as the package manager and Vitest on Node for stable
+tests, while using Bun for fast local development and tooling:
+
+- Primary (CI/stable): `pnpm build`, `pnpm test`, `pnpm test:ci`
+- Local convenience (Bun):
+  - `pnpm dev` – run the CLI from TypeScript via Bun
+  - `pnpm typecheck` – no-emit typechecking via `bunx tsc`
+  - `pnpm lint` / `pnpm lint:fix` – via `bunx eslint`
+  - `pnpm format` – via `bunx prettier`
+
+Notes:
+
+- Native addons (sharp, better-sqlite3) may build from source under Bun; Node
+  remains the primary test substrate.
+- Firecrawl SDK is compatible with Bun; the MCP server and tests remain on Node.
+
+## Architecture
+
+The pipeline follows a strict **4-stage architecture** with clear separation of
+concerns:
+
+```
+CSV/DB Exports
+      │
+      ├─────────────────┬──────────────────┐
+      ▼                 ▼                  ▼
+  Ingest-CSV       Ingest-DB         (Other sources)
+      │                 │                  │
+      └─────────────────┼──────────────────┘
+                        ▼
+              [Stage 1: Ingest]
+              Parse & normalize
+                        ▼
+             messages.*.ingested.json
+                        │
+                        ▼
+            [Stage 2: Normalize-Link]
+            Deduplicate, link replies/tapbacks
+                        ▼
+             messages.normalized.json
+                        │
+                        ▼
+              [Stage 3: Enrich-AI] ◄── Resumable & Incremental
+              Add AI enrichments
+                        ▼
+            messages.enriched.json
+                        │
+                        ▼
+             [Stage 4: Render-Markdown]
+             Generate daily files
+                        ▼
+              timeline/*.md (output)
+```
+
+### Stage 1: Ingest
+
+Extracts messages from CSV or SQLite database and normalizes to a unified
+schema.
+
+**Responsibilities:**
+
+- Parse rows with field mapping (handle CSV/DB dialect differences)
+- Convert dates (CSV UTC → ISO 8601, Apple epoch → ISO 8601)
+- Split rows into `text`/`media`/`notification`/`tapback` messages
+- Resolve attachment paths to absolute paths when possible
+- Create stable part GUIDs for multi-attachment DB messages:
+  `p:<index>/<original_guid>`
+- Preserve source metadata (CSV vs DB origin)
+
+**Input:** iMazing CSV or Messages.app SQLite database **Output:** Normalized
+`Message[]` in JSON envelope with metadata
+
+### Stage 2: Normalize-Link
+
+Merges multiple sources, deduplicates, links replies/tapbacks, and validates
+schema.
+
+**Responsibilities:**
+
+- Link replies to parents:
+  - Primary: DB `association_guid` (database-native association)
+  - Fallback: Heuristics (±30s timestamp proximity, text similarity, sender
+    difference)
+- Link tapbacks (emoji reactions) to message parts
+- Deduplicate across CSV/DB sources:
+  - Exact GUID matching (primary)
+  - Content equivalence (fuzzy text match, same sender, same timestamp)
+- Prefer DB-sourced data in conflicts (DB is authoritative for timestamps,
+  handles, etc.)
+- Enforce schema via Zod validation (camelCase, type correctness)
+
+**Algorithm Complexity:** O(n log n) for deduplication with GUID indexing
+
+**Input:** One or both ingest outputs **Output:** Merged, deduplicated, linked
+`messages.normalized.json`
+
+### Stage 3: Enrich-AI
+
+Augments messages with AI-powered analysis. Fully resumable and idempotent.
+
+**Responsibilities:**
+
+- Image analysis:
+  - Convert HEIC/TIFF to JPG preview (cached by filename)
+  - Gemini Vision API: structured prompt for caption + summary
+- Audio transcription:
+  - Structured prompt requesting timestamps and speaker labels
+  - Handles long audio with chunking (streaming for >10min files)
+- PDF summarization (key points extraction)
+- Link enrichment:
+  - Firecrawl for full web scraping
+  - Provider-specific fallbacks (YouTube, Spotify, Twitter, Instagram)
+  - Generic HTML meta tag fallback
+  - Graceful degradation (never crashes, stores error in enrichment)
+- Idempotent processing (skip if enrichment kind already exists)
+- Checkpointing (save progress every N items)
+- Resumable (load checkpoint, verify config hash, continue from last index)
+- Rate limiting (jittered backoff for API limits)
+- Incremental mode (process only new message GUIDs vs prior state)
+
+**Idempotency Key:** `(message.media.id, enrichment.kind)`
+
+**Input:** `messages.normalized.json`, optional checkpoint/state files
+**Output:** `messages.enriched.json` with populated `media.enrichment[]` arrays
+
+### Stage 4: Render-Markdown
+
+Generates deterministic daily markdown files organized by date and time-of-day.
+
+**Responsibilities:**
+
+- Group messages by calendar date
+- Sub-group by time-of-day sections (Morning 00:00-11:59, Afternoon 12:00-17:59,
+  Evening 18:00-23:59)
+- Render each message with:
+  - Timestamp anchor for deep linking
+  - Sender name / "Me" indicator
+  - Message text or media preview
+  - Enrichments (image captions, transcriptions, link contexts) as formatted
+    blockquotes
+- Render replies as nested blockquotes (up to configurable depth)
+- Render tapbacks as emoji reactions (❤️ for "loved", etc.)
+- Deterministic sorting by `(date, guid)` for reproducibility
+
+**Determinism:** Identical input → identical output. No randomization, stable
+key ordering.
+
+**Input:** `messages.enriched.json` **Output:** Daily markdown files
+(`timeline/YYYY-MM-DD.md`)
+
+## Message Schema
+
+The unified `Message` type represents all message kinds with a discriminated
+union:
+
+```typescript
+type Message = {
+  guid: string // Unique identifier
+  messageKind: 'text' | 'media' | 'tapback' | 'notification'
+  date: string // ISO 8601 with Z suffix (UTC)
+  isFromMe: boolean
+
+  // Optional fields by kind
+  text?: string // For text/notification messages
+  media?: MediaMeta // For media messages (see below)
+  tapback?: TapbackInfo // For tapback messages
+
+  // Linking
+  replyingTo?: ReplyInfo // Links to parent message GUID
+
+  // Metadata
+  service: string // SMS, iMessage, etc.
+  handle?: string // Phone number or Apple ID
+  senderName?: string // Display name
+  groupGuid?: string // For split messages, original DB GUID
+
+  // Preservation fields
+  subject?: string
+  isAudioMessage?: boolean
+  isDeleted?: boolean
+
+  // Provenance
+  sourceType?: 'csv' | 'db'
+  sourceMetadata?: Record<string, unknown>
+}
+
+type MediaMeta = {
+  id: string // Unique media ID
+  type: 'image' | 'audio' | 'pdf' | 'video' | 'document'
+  filename?: string
+  path?: string // Absolute path if file exists
+  mimeType?: string
+  size?: number
+  duration?: number // For audio/video in seconds
+  enrichment?: MediaEnrichment[] // AI analysis results
+  provenance?: {
+    originalPath?: string
+    source: 'csv' | 'db'
+    lastSeen?: string
+  }
+}
+
+type MediaEnrichment = {
+  kind: 'image_analysis' | 'transcription' | 'pdf_summary' | 'link_context'
+  content: Record<string, unknown>
+  provider: string // 'gemini', 'firecrawl', etc.
+  model: string
+  version: string
+  createdAt: string // ISO 8601
+  error?: string // If enrichment failed
+}
+```
+
+All dates are **ISO 8601 with Z suffix** (UTC). See
+[Dates and Timezones](#dates-and-timezones) for conversion details.
+
+## CLI Commands
+
+### Main Pipeline Commands
+
+#### `ingest-csv`
+
+Import messages from iMazing CSV export.
+
+```bash
+pnpm cli ingest-csv \
+  -i messages.csv \
+  -o messages.csv.ingested.json \
+  -a ~/Library/Messages/Attachments \
+  -a /Volumes/Backup/old-attachments
+```
+
+**Options:**
+
+- `-i, --input <path>` - iMazing CSV file (required)
+- `-o, --output <path>` - Output JSON file (default:
+  `./messages.csv.ingested.json`)
+- `-a, --attachments <dirs...>` - Root directories containing media files
+
+#### `ingest-db`
+
+Extract messages from macOS Messages.app SQLite database.
+
+```bash
+pnpm cli ingest-db \
+  -i ~/Library/Messages/chat.db \
+  -o messages.db.ingested.json \
+  --contact john@example.com
+```
+
+**Options:**
+
+- `-i, --input <path>` - Messages.app database file (required)
+- `-o, --output <path>` - Output JSON file (default:
+  `./messages.db.ingested.json`)
+- `--contact <id>` - Filter by contact (phone or Apple ID)
+- `-a, --attachments <dirs...>` - Attachment root directories
+
+#### `normalize-link`
+
+Merge sources, deduplicate, link replies/tapbacks, and validate schema.
+
+```bash
+pnpm cli normalize-link \
+  -i messages.csv.ingested.json messages.db.ingested.json \
+  -o messages.normalized.json \
+  -m all
+```
+
+**Options:**
+
+- `-i, --input <paths...>` - Input JSON files (required, can specify multiple)
+- `-o, --output <path>` - Output JSON file (default:
+  `./messages.normalized.json`)
+- `-m, --merge-strategy <strategy>` - `exact` (GUID only) | `content` (content
+  equivalence) | `all` (both, default)
+
+#### `enrich-ai`
+
+Augment messages with AI analysis (images, audio, links). Resumable and
+incremental.
+
+```bash
+pnpm cli enrich-ai \
+  -i messages.normalized.json \
+  -o messages.enriched.json \
+  --resume \
+  --incremental \
+  --rate-limit 1000 \
+  --max-retries 3 \
+  --checkpoint-interval 100 \
+  --enable-vision --enable-audio --enable-links \
+  -v
+```
+
+**Options:**
+
+- `-i, --input <path>` - Input normalized JSON (required)
+- `-o, --output <path>` - Output JSON file (default: `./messages.enriched.json`)
+- `-c, --checkpoint-dir <path>` - Checkpoint directory (default:
+  `./.checkpoints`)
+- `--resume` - Resume from last checkpoint
+- `--incremental` - Only enrich messages new since last enrichment run
+- `--state-file <path>` - Path to incremental state file (default:
+  `./.imessage-state.json`)
+- `--reset-state` - Clear incremental state and enrich all messages
+- `--rate-limit <ms>` - Delay between API calls (default: 1000)
+- `--max-retries <n>` - Max retries on API errors (default: 3)
+- `--checkpoint-interval <n>` - Save checkpoint every N items (default: 100)
+- `--enable-vision` - Enable image analysis (default: true)
+- `--enable-audio` - Enable audio transcription (default: true)
+- `--enable-links` - Enable link enrichment (default: true)
+
+#### `render-markdown`
+
+Generate daily markdown files from enriched messages.
+
+```bash
+pnpm cli render-markdown \
+  -i messages.enriched.json \
+  -o ./timeline \
+  --group-by-time \
+  --nested-replies \
+  --max-nesting-depth 10 \
+  --start-date 2025-01-01 \
+  --end-date 2025-12-31
+```
+
+**Options:**
+
+- `-i, --input <path>` - Input enriched JSON (required)
+- `-o, --output <path>` - Output directory (default: `./timeline`)
+- `--group-by-time` - Group by Morning/Afternoon/Evening (default: true)
+- `--nested-replies` - Render replies as blockquotes (default: true)
+- `--max-nesting-depth <n>` - Max blockquote nesting depth (default: 10)
+- `--start-date <YYYY-MM-DD>` - Filter messages from this date
+- `--end-date <YYYY-MM-DD>` - Filter messages until this date
+
+### Utility Commands
+
+#### `validate`
+
+Validate JSON file against Message schema.
+
+```bash
+pnpm cli validate -i messages.json [-q]
+```
+
+**Options:**
+
+- `-i, --input <path>` - JSON file to validate (required)
+- `-q, --quiet` - Suppress detailed error output
+
+**Output:** Exit code 0 on success, 1 on validation failure. Prints summary
+stats.
+
+#### `stats`
+
+Show statistics about a message file.
+
+```bash
+pnpm cli stats -i messages.json [-v]
+```
+
+**Options:**
+
+- `-i, --input <path>` - JSON file (required)
+- `-v, --verbose` - Show per-kind breakdown
+
+**Output:** Message count, breakdown by `messageKind`, date range, attachment
+count, etc.
+
+#### `doctor`
+
+Run system diagnostics.
+
+```bash
+pnpm cli doctor [-v]
+```
+
+**Checks:**
+
+- Node.js version (22+)
+- Dependencies (pnpm packages)
+- Config file exists and is readable
+- API keys present (GEMINI_API_KEY, FIRECRAWL_API_KEY)
+- Attachment directories accessible
+- Write permissions to output directories
+
+#### `init`
+
+Generate starter configuration file.
+
+```bash
+pnpm cli init [-f json|yaml] [--force] [-o custom-path.yaml]
+```
+
+**Options:**
+
+- `-f, --format <format>` - `json` or `yaml` (default: yaml)
+- `--force` - Overwrite existing config
+- `-o, --output <path>` - Custom config path
+
+## Configuration
+
+Configuration can be provided via `imessage-config.yaml` or
+`imessage-config.json`. Create with `pnpm cli init` or manually:
+
+```yaml
+version: '1.0'
+
+# Attachment directories to search for media files
+attachmentRoots:
+  - ~/Library/Messages/Attachments
+  - /Volumes/Backup/old-attachments
+
+# Google Gemini API configuration
+gemini:
+  apiKey: ${GEMINI_API_KEY} # Loaded from environment
+  model: gemini-1.5-pro # Recommended model
+  rateLimitDelay: 1000 # Milliseconds between requests
+  maxRetries: 3 # Retry failed API calls
+
+# Firecrawl (link enrichment) configuration
+firecrawl:
+  apiKey: ${FIRECRAWL_API_KEY} # Optional, for link context
+  enabled: true
+
+# Enrichment settings
+enrichment:
+  enableVisionAnalysis: true # Image captions/summaries
+  enableAudioTranscription: true # Audio transcription
+  enableLinkEnrichment: true # Link context extraction
+  imageCacheDir: ./.cache/images # Preview cache location
+  checkpointInterval: 100 # Items per checkpoint
+  forceRefresh: false # Re-enrich existing
+
+# Rendering settings
+render:
+  groupByTimeOfDay: true # Morning/Afternoon/Evening sections
+  renderRepliesAsNested: true # Blockquote threading
+  renderTapbacksAsEmoji: true # ❤️ instead of text
+  maxNestingDepth: 10 # Max blockquote levels
+```
+
+**Environment Variables:**
+
+- `GEMINI_API_KEY` - Google Gemini API key (required for enrichment)
+- `FIRECRAWL_API_KEY` - Firecrawl API key (optional, for link enrichment)
+- `TF_BUILD` - Set by CI systems (enables test reporters)
+
+**Config Loading:**
+
+- Looks for `imessage-config.yaml` or `imessage-config.json` in current
+  directory
+- Supports environment variable expansion: `${VARIABLE_NAME}`
+- CLI `--config` flag overrides default path
+
+## Data Flows & Examples
+
+### Example 1: Single Source (CSV Only)
+
+```bash
+# Ingest CSV
+pnpm cli ingest-csv -i messages.csv -o messages.ingested.json
+
+# Normalize (single source, minimal work)
+pnpm cli normalize-link -i messages.ingested.json -o messages.normalized.json
+
+# Enrich (first time, all messages)
+pnpm cli enrich-ai -i messages.normalized.json -o messages.enriched.json
+
+# Render
+pnpm cli render-markdown -i messages.enriched.json -o ./timeline
+```
+
+### Example 2: Dual Source with Incremental Enrichment
+
+```bash
+# Ingest both sources
+pnpm cli ingest-csv -i messages.csv -o messages.csv.ingested.json
+pnpm cli ingest-db -i ~/Library/Messages/chat.db -o messages.db.ingested.json
+
+# Normalize and merge
+pnpm cli normalize-link \
+  -i messages.csv.ingested.json messages.db.ingested.json \
+  -o messages.normalized.json
+
+# First enrichment
+pnpm cli enrich-ai \
+  -i messages.normalized.json \
+  -o messages.enriched.json \
+  --checkpoint-interval 100
+
+# Later: new messages added, re-run incrementally
+pnpm cli enrich-ai \
+  -i messages.normalized.json \
+  -o messages.enriched.json \
+  --incremental \
+  --resume
+```
+
+### Example 3: Resuming from Crash
+
+```bash
+# Enrichment stops mid-way (power loss, API timeout, etc.)
+# Checkpoint saved: .checkpoints/enrich-checkpoint-abc123def.json
+
+# Resume from checkpoint
+pnpm cli enrich-ai \
+  -i messages.normalized.json \
+  -o messages.enriched.json \
+  --resume
+# → Continues from last processed index automatically
+```
+
+## Dates and Timezones
+
+All dates in JSON outputs are **ISO 8601 UTC with Z suffix** (e.g.,
+`2025-10-26T14:30:45.000Z`).
+
+### CSV Import
+
+- iMazing CSV format: `MM/DD/YYYY, HH:MM:SS` (local timezone, interpreted as
+  UTC)
+- Converted to ISO 8601 with Z suffix
+
+### Database Import
+
+- Apple epoch: Seconds since 2001-01-01 00:00:00 UTC
+- Formula: `ISO = (appleSeconds + 978307200) * 1000` (convert to milliseconds,
+  then to ISO)
+- Result: ISO 8601 with Z suffix
+
+### Markdown Rendering
+
+- Timestamps displayed in UTC
+- Grouped by calendar date (UTC)
+- To display in local timezone, render the timestamp differently in
+  post-processing
+
+## Idempotency and Determinism
+
+### Idempotent Enrichment
+
+Enrichment is **idempotent** by design:
+
+- Check if `enrichment.kind` already exists for a message
+- Skip if present (already enriched)
+- Use `--force-refresh` to re-enrich specific kinds
+
+```bash
+# First run: enrich all
+pnpm cli enrich-ai -i messages.normalized.json -o messages.enriched.json
+
+# Later: add new enrichment kind (e.g., link context)
+# Existing image/audio enrichments preserved, new kind added
+pnpm cli enrich-ai -i messages.normalized.json -o messages.enriched.json \
+  --enable-links  # Other kinds disabled
+```
+
+### Deterministic Rendering
+
+Markdown output is **fully deterministic**:
+
+- Messages sorted by `(date, guid)` before rendering
+- Enrichments sorted by kind within each message
+- JSON keys in sorted order
+- No randomization or time-dependent output
+
+This means: `sha256(messages.enriched.json) → sha256(timeline/*.md)` is
+consistent across runs.
+
+### Checkpoint Consistency
+
+Checkpoints include config hash verification:
+
+- Each checkpoint stores SHA256 of enrichment config
+- Resume only if config unchanged
+- Detects breaking changes (API key updates, disable/enable analysis modes)
+
+## Performance & Optimization
+
+### Concurrency
+
+- **Ingest** (Stage 1): Single-threaded, fast (CSV parsing ~10k msgs/s)
+- **Normalize-Link** (Stage 2): Single-threaded, O(n log n) complexity (~1k
+  msgs/s for dedup)
+- **Enrich-AI** (Stage 3): API call bound, respectful rate limiting (1-5
+  msgs/min depending on Gemini quota)
+- **Render** (Stage 4): Single-threaded, fast (~10k msgs/s)
+
+### Memory
+
+- **Streaming where possible**: Large JSON files loaded once into memory
+- **Checkpoint interval**: Default 100 items keeps memory bounded
+- **Image cache**: Reuses converted previews by filename
+
+### Cost Optimization
+
+- **Incremental mode**: Only enrich new messages (~80% cost reduction for mature
+  datasets)
+- **Selective enrichment**: Enable/disable analysis modes (`--enable-vision`,
+  etc.)
+- **Image caching**: Preview conversion cached by filename (avoid re-processing)
+- **Fallback chain**: Use Firecrawl fallback before provider-specific parsing
+  (reduce API calls)
+
+### Suggested Workflow
+
+```bash
+# Initial run (expensive, one-time)
+pnpm cli enrich-ai -i normalized.json -o enriched.json --checkpoint-interval 100
+
+# Later: weekly incremental updates (cheap)
+pnpm cli enrich-ai -i normalized.json -o enriched.json --incremental --resume
+
+# Yearly: full re-enrichment with new models
+pnpm cli enrich-ai -i normalized.json -o enriched.json --force-refresh
+```
+
+## Advanced Usage
+
+### Merging Multiple CSV/DB Exports
+
+Combine multiple conversations into a single timeline:
+
+```bash
+# Ingest each conversation separately
+pnpm cli ingest-csv -i chat-with-alice.csv -o alice.ingested.json
+pnpm cli ingest-csv -i chat-with-bob.csv -o bob.ingested.json
+
+# Merge into single normalized file
+pnpm cli normalize-link \
+  -i alice.ingested.json bob.ingested.json \
+  -o messages.normalized.json
+
+# Enrich and render as single timeline
+pnpm cli enrich-ai -i messages.normalized.json -o messages.enriched.json
+pnpm cli render-markdown -i messages.enriched.json -o ./timeline
+```
+
+### Selective Date Range Rendering
+
+Render only recent messages:
+
+```bash
+pnpm cli render-markdown \
+  -i messages.enriched.json \
+  -o ./timeline \
+  --start-date 2025-10-01 \
+  --end-date 2025-10-31
+```
+
+### Upgrading Enrichment Models
+
+Re-enrich with newer Gemini models:
+
+```bash
+# Update config with new model
+# imessage-config.yaml:
+# gemini:
+#   model: gemini-2-flash  # (hypothetical future model)
+
+# Re-enrich with new model
+pnpm cli enrich-ai \
+  -i messages.normalized.json \
+  -o messages.enriched.json \
+  --force-refresh
+```
+
+## Testing
+
+See our testing guide for configuration rationale and patterns:
+`docs/testing-best-practices.md`.
+
+The project includes 50+ test files covering:
+
+- Schema validation (happy path + invariant violations)
+- CSV/DB ingestion (parsing, path resolution, date conversion)
+- Linking algorithms (reply matching, tapback association)
+- Deduplication (GUID matching, content equivalence)
+- Enrichment idempotency (skip logic, force-refresh)
+- Checkpoint recovery (save/load, config hash verification)
+- Rendering (grouping, sorting, determinism)
+
+### Run Tests
+
+```bash
+# All tests
+pnpm test
+
+# Watch mode (re-run on file change)
+pnpm test --watch
+
+# Coverage report
+pnpm test:coverage
+```
+
+### Coverage
+
+Maintained at **70%+ branch coverage**. Critical paths (linking, dedup,
+enrichment) at 95%+.
+
+## Contributing
+
+Contributions welcome! Please:
+
+1. Fork the repository
+2. Create a feature branch (`git checkout -b feature/amazing-feature`)
+3. Add tests for new functionality
+4. Ensure tests pass: `pnpm test`
+5. Format code: `pnpm format`
+6. Lint: `pnpm lint`
+7. **Create a changeset**: `pnpm version:gen` (for user-facing changes)
+8. Commit with semantic message: `git commit -m "feat: add amazing feature"`
+   - ✅ Hooks automatically validate commit format and code quality
+9. Push and create a pull request
+
+### Development Setup
+
+```bash
+# Install with dev dependencies
+pnpm install
+
+# Start watch mode (auto-rebuild TypeScript)
+pnpm watch
+
+# Run tests in watch mode during development
+pnpm test --watch
+
+# Check code quality before committing
+pnpm quality-check
+```
+
+### Branch protection and local push guard
+
+To configure GitHub branch protection on `main` using our aggregate gate:
+
+```bash
+bash scripts/setup-branch-protection.sh nathanvale chatline main
+```
+
+This script enables auto-merge, enforces required checks (PR quality / gate,
+Commitlint / commitlint, PR Title Lint / lint), requires signed commits, and
+keeps history linear.
+
+Local safeguard: direct pushes to `main`/`master` are blocked by a Husky
+pre-push hook. Create a feature branch and open a PR instead.
+
+If you absolutely must override locally (not recommended):
+
+```bash
+ALLOW_PUSH_PROTECTED=1 git push
+```
+
+### Code Style
+
+- **TypeScript**: Strict mode, no `any`
+- **Formatting**: Prettier with 80-char line limit
+- **Linting**: ESLint with recommended rules
+- **Testing**: Vitest with 70%+ coverage threshold
+- **Commits**: Conventional commits (feat:, fix:, docs:, etc.)
+  - See [Automated Release Workflow](./docs/automated-release-workflow.md) for
+    commit format guide
+
+### Release Process
+
+This project uses **automated releases** with Changesets:
+
+- **Create changeset** for user-facing changes: `pnpm version:gen`
+- **Commit messages** validated automatically via Husky + commitlint
+- **CI/CD** creates "Version Packages" PR when changesets are merged
+- **Publishing** happens automatically when version PR is merged
+
+📚 **Full documentation:**
+
+- **[Automated Release Workflow](./docs/automated-release-workflow.md)** - Main
+  release process
+- **[Pre-Release Guide](./docs/pre-release-guide.md)** - Canary, beta, and RC
+  releases
+
+### Release Channels
+
+We support prerelease channels for fast feedback and safe promotion:
+
+- `next` for early adopters (canary builds)
+- `beta` for feature-complete testing
+- `rc` for release candidates
+- `canary` snapshots for experimental builds
+- `alpha` for automated nightly snapshots
+
+**Quick commands:**
+
+```bash
+# Publish quick snapshot (only when NOT in pre-mode)
+pnpm release:snapshot:canary
+
+# Publish versioned pre-release (when in pre-mode)
+pnpm changeset              # Create changeset
+pnpm changeset version      # Version as 0.0.1-next.0
+pnpm publish:pre            # Publish to @next tag
+
+# Enter/exit pre-release mode
+gh workflow run pre-mode.yml -f action=enter -f channel=next
+gh workflow run pre-mode.yml -f action=exit -f channel=next
+```
+
+> ⚠️ **Note:** Snapshot releases (`pnpm release:snapshot:canary`) only work when
+> NOT in pre-release mode. When in pre-mode, use versioned pre-releases instead.
+
+📚 **See the detailed guides:**
+
+- **[Pre-Release Guide](./docs/pre-release-guide.md)** - Step-by-step publishing
+  instructions
+- **[Release Channels Strategy](./docs/release-channels.md)** - Architecture and
+  promotion flows
+
+### Package Hygiene
+
+We enforce a tight publish surface and solid metadata:
+
+- Validated with publint and AreTheTypesWrong
+- Minimal tarball via `files` whitelist
+- Provenance enabled for trusted builds
+
+See the full checklist and how to run the checks:
+[Package Hygiene & Metadata Quality](./docs/package-hygiene.md)
+
+### Prettier formatting
+
+We use a minimal, opinionated Prettier setup:
+
+- Global 80-char width, trailing commas, single quotes, no semicolons
+- Deterministic JSON sorting via plugin
+- Non-mutating check for CI/local validation
+
+Docs and rationale:
+[Prettier Best Practices & Formatting Strategy](./docs/prettier-best-practices.md)
+
+## Troubleshooting
+
+### "API rate limit exceeded"
+
+**Solution:** Increase `--rate-limit` delay
+
+```bash
+pnpm cli enrich-ai -i messages.normalized.json -o enriched.json --rate-limit 2000
+```
+
+### "Checkpoint config hash mismatch"
+
+**Cause:** Changed enrichment config (API key, enable/disable analysis)
+**Solution:** Use `--reset-state` to clear or manually delete
+`.imessage-state.json`
+
+```bash
+pnpm cli enrich-ai -i messages.normalized.json -o enriched.json --reset-state
+```
+
+### "Attachment paths not resolved"
+
+**Cause:** Media file not found in attachment directories **Check:**
+
+1. Verify path in config (`attachmentRoots`)
+2. Check file exists on disk
+3. Check file permissions **Result:** Path stored as filename with provenance
+   metadata
+
+### "Validation errors in normalized.json"
+
+**Debug:**
+
+```bash
+pnpm cli validate -i messages.normalized.json -v
+# Shows which fields failed validation
+```
+
+**Common causes:**
+
+- Missing `messageKind` field
+- Date not in ISO 8601 UTC format
+- Inconsistent data types (string vs number)
+
+Run `pnpm cli doctor` for system-level diagnostics.
+
+## FAQ
+
+**Q: Can I use this on Linux/Windows?** A: CSV ingestion works everywhere.
+Database ingestion requires macOS (to access Messages.app). You can export from
+macOS and process on other systems.
+
+**Q: How much storage do the outputs take?** A: Enriched JSON is typically 2-3x
+original normalized JSON (due to enrichment data). Markdown files are 1-2x
+enriched JSON. A 1000-message conversation: ~5-10MB JSON, ~10-20MB markdown.
+
+**Q: Can I re-use enriched.json if I change the render config?** A: Yes!
+Rendering is deterministic and config-independent. Change render settings
+(grouping, nesting depth) and re-render without re-enriching.
+
+**Q: What if I don't have API keys?** A: Enrichment skips (messages remain
+as-is). Set `--enable-vision false --enable-audio false --enable-links false` to
+disable. Rendering still works perfectly without enrichment.
+
+**Q: How do I update my timeline when new messages arrive?** A: Re-export from
+Messages.app/iMazing, then run the full pipeline OR use `--incremental --resume`
+to process only new messages (80%+ faster).
+
+**Q: Is my data private?** A: Yes. All processing is local. API calls to
+Gemini/Firecrawl are necessary for enrichment but never persist to artifacts. No
+data retained after processing. Set API keys via environment variables (not in
+config files).
+
+## Technical Details
+
+### Schema Invariants
+
+Messages enforce cross-field constraints via Zod `superRefine()`:
+
+- `messageKind='media'` → `media` field must exist and be complete
+- `messageKind='tapback'` → `tapback` field must exist
+- `messageKind='text'|'notification'` → may have text, must not have
+  media/tapback
+- All dates must be ISO 8601 with Z suffix
+
+### Linking Heuristics
+
+Reply linking uses a confidence-scoring algorithm:
+
+1. Check DB association (if present, use immediately)
+2. Search ±30s timestamp window
+3. Score candidates:
+   - Timestamp distance: closer = higher score
+   - Text similarity: matching keywords = higher score
+   - Sender difference: different person = higher score (likely replying)
+4. Select highest score (or log as ambiguous if tie)
+
+### Deduplication Strategy
+
+CSV/DB deduplication uses a multi-pass approach:
+
+1. Exact GUID matching (primary)
+2. Content equivalence (fuzzy text + same sender + same timestamp)
+3. Prefer DB values in conflicts (authoritiveness)
+4. Sort by GUID for determinism
+
+### Idempotency Design
+
+Enrichment is idempotent via kind-based deduplication:
+
+- Each enrichment entry has a `kind` (e.g., `'image_analysis'`,
+  `'transcription'`)
+- Check if `kind` already exists before enriching
+- `forceRefresh` replaces specific kind (preserves others)
+- Result: Safe to re-run without duplicating enrichments
+
+## Roadmap
+
+- [ ] Support for WhatsApp, Telegram exports
+- [ ] Batch API calls to reduce Gemini quota usage
+- [ ] Vector embeddings for similarity search
+- [ ] Web UI for browsing/searching timeline
+- [ ] Obsidian plugin for live sync
+- [ ] Self-hosted LLM support (Ollama, etc.)
+- [ ] Photo gallery view alongside markdown
+- [ ] Sentiment analysis and conversation metrics
+- [ ] Anonymous mode (redact PII)
+
+## License
+
+MIT © 2025
+
+See [LICENSE](LICENSE) file for full text.
+
+## Related Projects
+
+- [iMazing](https://imazing.com/) - CSV export source
+- [Firecrawl](https://www.firecrawl.dev/) - Link enrichment API
+- [Google Gemini](https://ai.google.dev/) - Image/audio analysis API
+- [Obsidian](https://obsidian.md/) - Markdown vault system
+
+## Contact & Support
+
+- **Issues & Bugs**:
+  [GitHub Issues](https://github.com/yourusername/chatline/issues)
+- **Discussions**:
+  [GitHub Discussions](https://github.com/yourusername/chatline/discussions)
+- **Email**: support@example.com (replace with actual contact)
+
+---
+
+**Enjoying iMessage Timeline?** Please star ⭐ the repo and share with friends!