@libredb/studio 0.9.7

package/docs/backlogs/003-QUERY_TIME_MACHINE.md

@@ -0,0 +1,183 @@
+# Query Time Machine
+
+## Overview
+Compare query results across different points in time. Answer questions like "What did this data look like yesterday vs today?"
+
+## Problem Statement
+Data analysts frequently need to understand how data has changed over time:
+- "Why did our sales drop? What changed since last week?"
+- "This customer had 5 orders yesterday, now shows 3. What happened?"
+- "Compare today's active users with last month"
+
+Currently, users must:
+- Manually save query results to files/spreadsheets
+- Re-run queries and manually compare
+- Rely on memory or external documentation
+
+## Proposed Solution
+Automatic query result snapshots with visual diff comparison.
+
+## Features
+
+### 1. Automatic Snapshots
+- Save query results with timestamp automatically
+- Configurable retention (7 days, 30 days, custom)
+- Storage optimization (compress, deduplicate)
+- Manual "Pin" option for important snapshots
+
+### 2. Time Selector UI
+```
+┌─────────────────────────────────────────────────────────────┐
+│ Query Results                    [Today ▼] vs [Yesterday ▼] │
+├─────────────────────────────────────────────────────────────┤
+│ Timeline: ───●────●────●────●────●───                       │
+│          Dec 20  21   22   23   24                          │
+└─────────────────────────────────────────────────────────────┘
+```
+- Dropdown to select comparison dates
+- Visual timeline with snapshot points
+- Quick presets: "vs Yesterday", "vs Last Week", "vs Last Month"
+
+### 3. Diff Visualization
+```
+┌─────────────────────────────────────────────────────────────┐
+│ Changes Summary: +12 new rows, -3 removed, ~5 modified      │
+├─────────────────────────────────────────────────────────────┤
+│   name          │ orders (Dec 23) │ orders (Dec 24) │ diff  │
+├─────────────────┼─────────────────┼─────────────────┼───────┤
+│ + Alice Smith   │ -               │ 15              │ NEW   │
+│ ~ John Doe      │ 42              │ 45              │ +3    │
+│ ~ Jane Wilson   │ 38              │ 35              │ -3    │
+│ - Bob Johnson   │ 12              │ -               │ DEL   │
+└─────────────────────────────────────────────────────────────┘
+
+Legend: + Added  ~ Modified  - Removed
+```
+- Side-by-side comparison
+- Inline diff with color coding
+- Summary statistics (added/removed/modified counts)
+- Filter: "Show only changes"
+
+### 4. Trend Analysis
+- Mini sparkline showing value changes over time
+- "This value increased 23% over the last 7 snapshots"
+- Anomaly detection: "Unusual drop detected on Dec 22"
+
+### 5. Query History Integration
+- Link snapshots to specific query executions
+- "Show me all snapshots for this query"
+- Compare results from different query versions
+
+## Technical Considerations
+
+### Storage Strategy
+```typescript
+interface QuerySnapshot {
+  id: string;
+  queryHash: string;           // Hash of the SQL query
+  connectionId: string;
+  timestamp: Date;
+  rowCount: number;
+  checksum: string;            // For quick equality check
+  data: CompressedData;        // Compressed row data
+  metadata: {
+    executionTime: number;
+    fields: string[];
+  };
+}
+```
+
+### Storage Options
+1. **LocalStorage/IndexedDB** - Client-side, limited capacity
+2. **Server-side SQLite** - Dedicated snapshot database
+3. **Cloud Storage** - S3/GCS for large datasets
+4. **Hybrid** - Recent in browser, older in cloud
+
+### Diff Algorithm
+- Use row primary key for matching
+- Hash-based comparison for quick equality check
+- Deep comparison for modified detection
+- Efficient diff for large datasets (streaming)
+
+### API Endpoints
+```
+GET /api/snapshots?queryHash=xxx
+POST /api/snapshots
+GET /api/snapshots/:id
+DELETE /api/snapshots/:id
+GET /api/snapshots/diff?from=id1&to=id2
+```
+
+## UI Components
+
+### New Components
+- `TimelineSelector.tsx` - Visual timeline with snapshot points
+- `SnapshotDiff.tsx` - Side-by-side comparison view
+- `SnapshotManager.tsx` - List and manage saved snapshots
+- `TrendSparkline.tsx` - Mini trend visualization
+
+### Integration Points
+- Results toolbar: "Compare with..." button
+- Query history: "View snapshots" action
+- New bottom panel tab: "Time Machine"
+
+## User Flow
+
+```
+1. User runs query
+   ↓
+2. System auto-saves snapshot (if enabled)
+   ↓
+3. User clicks "Compare with..." or "Time Machine" tab
+   ↓
+4. User selects comparison date/snapshot
+   ↓
+5. System shows diff view with changes highlighted
+   ↓
+6. User can drill down into specific changes
+```
+
+## Configuration Options
+- Auto-snapshot: On/Off
+- Retention period: 7/30/90 days
+- Max snapshots per query: 10/50/100
+- Storage limit: 50MB/200MB/1GB
+- Compression level: Low/Medium/High
+
+## Edge Cases
+- Query with no primary key (use row index)
+- Schema changes between snapshots
+- Very large result sets (pagination/sampling)
+- Identical results (show "No changes" state)
+
+## Empty States
+- "No snapshots yet. Run a query to create your first snapshot."
+- "This is the first snapshot. Run the query again later to compare."
+- "No changes between selected snapshots."
+
+## Acceptance Criteria
+- [ ] Query results are automatically saved as snapshots
+- [ ] User can select two snapshots to compare
+- [ ] Diff view shows added, removed, and modified rows
+- [ ] Changes are color-coded for easy identification
+- [ ] Summary shows counts of each change type
+- [ ] User can filter to show only changes
+- [ ] Snapshots can be manually pinned/deleted
+- [ ] Storage limits are enforced
+- [ ] Works with PostgreSQL, MySQL, SQLite
+
+## Dependencies
+- Query execution system
+- Storage system (IndexedDB or server-side)
+- Diff algorithm implementation
+
+## Estimated Effort
+High complexity
+
+## Priority
+P2 - Differentiating feature
+
+## Related Features
+- Query History (existing)
+- Data Visualization (existing)
+- AI Data Storyteller (planned)

package/docs/backlogs/004-AI_DATA_STORYTELLER.md

@@ -0,0 +1,292 @@
+# AI Data Storyteller
+
+## Overview
+Transform raw query results into meaningful narratives, insights, and executive summaries using AI. Move beyond "what" the data shows to "why it matters."
+
+## Problem Statement
+Data analysts spend significant time:
+- Interpreting query results manually
+- Writing reports explaining data findings
+- Identifying trends and anomalies by eye
+- Translating technical data into business language
+
+Non-technical stakeholders struggle to:
+- Understand raw data tables
+- Identify what's important in large datasets
+- Make decisions based on numbers alone
+
+## Proposed Solution
+AI-powered data interpretation that automatically generates:
+- Natural language summaries
+- Key insights and anomalies
+- Trend analysis
+- Executive-ready reports
+
+## Features
+
+### 1. One-Click Insights
+```
+┌─────────────────────────────────────────────────────────────┐
+│ Results (1,247 rows)                    [✨ Tell me a story]│
+├─────────────────────────────────────────────────────────────┤
+│                                                             │
+│  ┌─────────────────────────────────────────────────────┐   │
+│  │ 📊 Data Story                                        │   │
+│  │                                                      │   │
+│  │ Key Findings:                                        │   │
+│  │ • Sales increased 23% compared to last month        │   │
+│  │ • Top performer: Electronics category (+45%)        │   │
+│  │ • Warning: Returns in Clothing up 12%               │   │
+│  │                                                      │   │
+│  │ Anomalies Detected:                                  │   │
+│  │ • Dec 15: Unusual spike in orders (Black Friday?)   │   │
+│  │ • Region "West" underperforming by 2 std deviations │   │
+│  │                                                      │   │
+│  │ Recommendation:                                      │   │
+│  │ "Investigate Clothing returns - may indicate        │   │
+│  │  quality or sizing issues"                          │   │
+│  └─────────────────────────────────────────────────────┘   │
+└─────────────────────────────────────────────────────────────┘
+```
+
+### 2. Smart Question Interface
+Instead of writing SQL, ask questions in natural language:
+```
+User: "What's interesting about this data?"
+AI: "I notice three key patterns..."
+
+User: "Why did sales drop in March?"
+AI: "Looking at the data, the drop correlates with..."
+
+User: "Summarize this for my manager"
+AI: "Executive Summary: Q4 performance exceeded targets..."
+```
+
+### 3. Automatic Analysis Types
+
+#### Statistical Summary
+- Min, max, mean, median, std deviation
+- Distribution analysis
+- Correlation between columns
+
+#### Trend Detection
+- Time-series pattern recognition
+- Seasonality identification
+- Growth rate calculations
+
+#### Anomaly Detection
+- Outlier identification
+- Unexpected patterns
+- Missing data patterns
+
+#### Comparative Analysis
+- Group comparisons
+- Period-over-period changes
+- Benchmark comparisons
+
+### 4. Report Generation
+```
+┌─────────────────────────────────────────────────────────────┐
+│ Generate Report                                             │
+├─────────────────────────────────────────────────────────────┤
+│ Format:    [Executive Summary ▼]                            │
+│ Audience:  [Non-Technical ▼]                                │
+│ Include:   [✓] Charts  [✓] Key Metrics  [ ] Raw Data       │
+│ Tone:      [Professional ▼]                                 │
+│                                                             │
+│                              [Generate PDF] [Copy Markdown] │
+└─────────────────────────────────────────────────────────────┘
+```
+
+Report templates:
+- Executive Summary (1 page, high-level)
+- Detailed Analysis (comprehensive)
+- Technical Report (for data teams)
+- Presentation Slides (bullet points + charts)
+
+### 5. Interactive Exploration
+```
+"Tell me more about the Electronics category"
+     ↓
+AI drills down and shows subcategory breakdown
+
+"Compare this with last year"
+     ↓
+AI generates year-over-year comparison
+
+"What should we do about this?"
+     ↓
+AI provides actionable recommendations
+```
+
+## Technical Considerations
+
+### AI Integration
+```typescript
+interface StorytellerRequest {
+  result: QueryResult;
+  schemaContext: TableSchema[];
+  queryContext: string;           // The SQL that generated this
+  analysisType: 'summary' | 'trends' | 'anomalies' | 'full';
+  audience: 'technical' | 'business' | 'executive';
+  previousContext?: string;       // For follow-up questions
+}
+
+interface StorytellerResponse {
+  summary: string;
+  keyFindings: Finding[];
+  anomalies: Anomaly[];
+  recommendations: string[];
+  charts: ChartSuggestion[];
+  confidence: number;
+}
+```
+
+### LLM Prompting Strategy
+1. **Context Building**
+   - Schema information (table/column names, types)
+   - Query intent (what user was looking for)
+   - Result statistics (row count, value ranges)
+
+2. **Analysis Prompt**
+   ```
+   You are a data analyst. Given this query result:
+   - Schema: [tables and columns]
+   - Query: [SQL]
+   - Sample data: [first 100 rows]
+   - Statistics: [aggregates]
+
+   Provide insights in this format:
+   1. Key Findings (3-5 bullet points)
+   2. Anomalies (if any)
+   3. Trends (if time-series)
+   4. Recommendations
+   ```
+
+3. **Audience Adaptation**
+   - Technical: Include SQL, statistical terms
+   - Business: Focus on metrics, KPIs
+   - Executive: High-level, actionable
+
+### Data Sampling for Large Results
+- First 100 rows + random sample of 100
+- Statistical aggregates (min, max, avg, percentiles)
+- Column value distributions
+- Never send full dataset to LLM
+
+### Streaming Response
+- Stream AI response for better UX
+- Show "Analyzing..." with progress
+- Typewriter effect for narrative
+
+## UI Components
+
+### New Components
+- `DataStoryPanel.tsx` - Main storyteller interface
+- `InsightCard.tsx` - Individual insight display
+- `AnomalyBadge.tsx` - Anomaly highlight component
+- `ReportGenerator.tsx` - Report export modal
+- `QuestionInput.tsx` - Natural language question input
+
+### Integration Points
+- Results toolbar: "Tell me a story" button
+- New bottom panel mode or floating panel
+- Context menu on columns: "Analyze this column"
+- Query editor: Natural language input mode
+
+## User Flow
+
+```
+1. User runs query, sees results
+   ↓
+2. Clicks "✨ Tell me a story" button
+   ↓
+3. AI analyzes data (streaming response)
+   ↓
+4. Insights panel shows findings
+   ↓
+5. User can ask follow-up questions
+   ↓
+6. User exports as report (optional)
+```
+
+## Example Outputs
+
+### For Sales Data
+```
+📊 Sales Analysis - December 2024
+
+Key Findings:
+• Total revenue: $1.2M (+18% vs November)
+• Best day: December 15 ($89K) - likely Black Friday effect
+• Top category: Electronics (42% of revenue)
+
+⚠️ Anomalies:
+• Unusually high returns in Clothing (12% vs 5% average)
+• Western region 23% below other regions
+
+📈 Trends:
+• Steady growth since October (+8% MoM)
+• Weekend sales 34% higher than weekdays
+
+💡 Recommendations:
+1. Investigate Clothing returns - possible sizing issue
+2. Review Western region - may need marketing push
+```
+
+### For User Analytics
+```
+📊 User Activity Report
+
+Key Findings:
+• 15,234 active users this week (+5%)
+• Average session: 12 minutes
+• Most active: Tuesday 2-4 PM
+
+⚠️ Concerns:
+• 23% bounce rate on mobile (desktop: 8%)
+• New user retention dropped to 34%
+
+💡 Recommendations:
+1. Prioritize mobile experience optimization
+2. Review onboarding flow for new users
+```
+
+## Configuration Options
+- Default analysis depth: Quick / Standard / Deep
+- Preferred language style: Casual / Professional / Academic
+- Auto-analyze on query: On / Off
+- Include chart suggestions: On / Off
+
+## Privacy & Security
+- Data sampling (never full dataset to LLM)
+- Option to disable for sensitive queries
+- No PII in prompts (mask if detected)
+- Local-only mode (Ollama) for sensitive data
+
+## Acceptance Criteria
+- [ ] "Tell me a story" button appears on results
+- [ ] AI generates natural language summary
+- [ ] Key findings are extracted and highlighted
+- [ ] Anomalies are detected and flagged
+- [ ] User can ask follow-up questions
+- [ ] Reports can be exported as PDF/Markdown
+- [ ] Analysis adapts to audience type
+- [ ] Streaming response for better UX
+- [ ] Works with existing LLM providers
+
+## Dependencies
+- LLM integration (existing)
+- Data Visualization (for chart suggestions)
+- Export functionality
+
+## Estimated Effort
+High complexity
+
+## Priority
+P1 - Core differentiator
+
+## Related Features
+- AI Query Assistant (existing)
+- Data Visualization (existing)
+- Query Time Machine (planned)