@opendash-ai/cli 0.1.0-beta.1 → 0.1.0-beta.2

package/src/intelligence/PHASE3-REFINED.md

@@ -1,257 +0,0 @@
-# Phase 3 Refined: Observable Analytical Behavior Metrics
-
-## Key Insight
-
-The benchmark must measure **what the user sees**, not internal execution.
-
-### Two Distinct Dimensions
-
-| Dimension | What It Measures | Benchmarked? |
-|-----------|------------------|--------------|
-| Internal Reasoning Quality | How the system thinks | NO |
-| External Observable Behavior | What the user sees | YES |
-
-## What Changed
-
-### Before (Internal Focus)
-- Confidence calibration values
-- Internal reasoning steps
-- Internal state transitions
-
-### After (Observable Focus)
-- Problem decomposition SHOWN to user
-- Workflow selection COMMUNICATED
-- Hypotheses PRESENTED with evidence
-- Data requirements LISTED
-- Statistical methods RECOMMENDED
-- KPI relationships EXPLAINED
-- Recommendations PRIORITIZED
-- Risks DISCLOSED
-
-## Benchmark Specification Format
-
-Each benchmark defines EXPECTED OBSERVABLE OUTPUTS:
-
-```json
-{
-  "benchmark_id": "retail-001",
-  "question": "Sales dropped by 18%",
-  "expected_observable_behavior": {
-    "problem_decomposition": {
-      "problem_type_shown": "diagnostic",
-      "stakeholders_mentioned": ["Sales", "Finance"],
-      "decision_context": "Operational"
-    },
-    "workflow_selection": {
-      "workflow_name_shown": "Root Cause Analysis",
-      "reasoning_provided": true,
-      "alternatives_mentioned": true
-    },
-    "hypotheses": {
-      "count_shown": "4-6",
-      "each_has_evidence": true,
-      "each_has_test_method": true,
-      "each_has_confidence": true
-    },
-    ...
-  }
-}
-```
-
-## Observable Behavior Metrics
-
-### 1. Problem Decomposition Visibility
-**Question**: Did the system SHOW the user how it decomposed the problem?
-
-| Observable | Score 10 | Score 0 |
-|------------|----------|---------|
-| Problem type shown | Explicitly stated | Hidden |
-| Stakeholders mentioned | Listed clearly | Not mentioned |
-| Decision context | Explained | Not explained |
-
-### 2. Workflow Selection Transparency
-**Question**: Did the system COMMUNICATE which workflow it chose and why?
-
-| Observable | Score 10 | Score 0 |
-|------------|----------|---------|
-| Workflow name shown | "Root Cause Analysis" | Hidden |
-| Reasoning provided | "Because problem is diagnostic" | No reasoning |
-| Alternatives mentioned | "Other options: EDA, Forecasting" | No alternatives |
-
-### 3. Hypothesis Presentation
-**Question**: Did the system PRESENT hypotheses with supporting evidence?
-
-| Observable | Score 10 | Score 0 |
-|------------|----------|---------|
-| Count shown | "5 hypotheses generated" | Hidden |
-| Each has evidence | Supporting/against listed | No evidence |
-| Each has test method | How to verify | No method |
-| Each has confidence | Medium/High/Low | No confidence |
-
-### 4. Data Requirements Disclosure
-**Question**: Did the system LIST what data is needed?
-
-| Observable | Score 10 | Score 0 |
-|------------|----------|---------|
-| Critical data listed | "Need marketing spend data" | Hidden |
-| Columns specified | "date, spend, impressions" | No columns |
-| Source identified | "From marketing platform" | No source |
-
-### 5. Statistical Method Communication
-**Question**: Did the system RECOMMEND appropriate methods?
-
-| Observable | Score 10 | Score 0 |
-|------------|----------|---------|
-| Methods named | "Time Series Decomposition" | Hidden |
-| Assumptions listed | "Requires stationarity" | No assumptions |
-| When to use explained | "Use for temporal patterns" | No explanation |
-
-### 6. KPI Relationship Explanation
-**Question**: Did the system EXPLAIN metric relationships?
-
-| Observable | Score 10 | Score 0 |
-|------------|----------|---------|
-| Equation shown | "Revenue = Traffic × Conversion × AOV" | Hidden |
-| Components identified | "3 components: Traffic, Conversion, AOV" | Not identified |
-| Mathematical insight | "AOV declined 15%, primary driver" | No insight |
-
-### 7. Recommendation Prioritization
-**Question**: Did the system PRIORITIZE recommendations?
-
-| Observable | Score 10 | Score 0 |
-|------------|----------|---------|
-| Count shown | "5 recommendations" | Hidden |
-| Prioritized | P1, P2, P3 labels | Not prioritized |
-| Impact assessed | "High impact: $1.2M" | No impact |
-| Effort assessed | "Low effort: 1 week" | No effort |
-| Risk assessed | "Medium risk" | No risk |
-
-### 8. Risk Disclosure
-**Question**: Did the system DISCLOS risks and limitations?
-
-| Observable | Score 10 | Score 0 |
-|------------|----------|---------|
-| Risks listed | "Risk: Seasonal factors" | Hidden |
-| Confidence levels | "Confidence: Medium" | No confidence |
-| Limitations acknowledged | "Limitation: Small sample" | No limitations |
-
-## Flow Score
-
-Measures what percentage of the designed workflow is visible to the user:
-
-```
-Flow Score = (Visible Steps / Total Steps) × 10
-
-Designed Flow:
-1. Problem Decomposition (SHOWN)
-2. Workflow Selection (SHOWN)
-3. Hypotheses (SHOWN)
-4. Data Requirements (SHOWN)
-5. Statistical Methods (SHOWN)
-6. KPI Reasoning (SHOWN)
-7. Recommendations (SHOWN)
-8. Risk Disclosure (SHOWN)
-
-If 7/8 steps visible:
-Flow Score = 7/8 × 10 = 8.75
-```
-
-## Pass/Fail Criteria
-
-### PASS if:
-1. **Observable Score ≥ 8.0** (user sees what they should)
-2. **Flow Score ≥ 8.0** (80% of steps visible)
-3. **No hidden critical steps** (workflow, hypotheses, recommendations visible)
-4. **Evidence provided** (hypotheses have supporting/against)
-5. **Risks disclosed** (limitations acknowledged)
-
-### FAIL if:
-1. Observable Score < 6.0
-2. Critical steps hidden
-3. No reasoning provided
-4. Risks not disclosed
-
-## Running Benchmarks
-
-### Command
-```bash
-cd packages/opencode
-bun test src/intelligence/analytics/benchmarks/observable-runner.ts
-```
-
-### Output
-```
-=== Observable Analytical Behavior Benchmark ===
-
-Benchmark: retail-001
-Question: "Sales dropped by 18%"
-
-Observable Behavior:
-  Problem Decomposition:    9/10 ✓
-  Workflow Selection:       10/10 ✓
-  Hypothesis Presentation:  8/10 ✓
-  Data Requirements:        7/10 ✓
-  Statistical Methods:      8/10 ✓
-  KPI Reasoning:            9/10 ✓
-  Recommendation Priority:  8/10 ✓
-  Risk Disclosure:          9/10 ✓
-  ─────────────────────────────
-  Overall Observable Score: 8.7/10
-  Flow Score:               10.0/10 (8/8 steps visible)
-  Passed:                   ✓ YES
-
-=== Summary ===
-Total Benchmarks: 2
-Average Observable Score: 8.7/10
-Pass Rate: 100%
-```
-
-## What This Proves
-
-### NOT Proved
-- Internal confidence values are correct
-- Internal reasoning is optimal
-- Internal state transitions work
-
-### PROVED
-- User SEES the workflow selection
-- User READS the hypotheses with evidence
-- User UNDERSTANDS the KPI relationships
-- User CAN ACT on the recommendations
-- User KNOWS the risks and limitations
-
-## Architecture Promise Validation
-
-The system promises this flow:
-```
-User Question
-    ↓
-Problem Decomposition (SHOWN)
-    ↓
-Workflow Selection (SHOWN)
-    ↓
-Hypotheses with Evidence (SHOWN)
-    ↓
-Data Requirements (SHOWN)
-    ↓
-Statistical Methods (SHOWN)
-    ↓
-KPI Reasoning (SHOWN)
-    ↓
-Recommendations (SHOWN)
-    ↓
-Risk Disclosure (SHOWN)
-```
-
-The benchmark VALIDATES that each step is:
-1. VISIBLE to user
-2. Has REASONING provided
-3. Connects to NEXT step
-4. Not SKIPPED
-5. Not HIDDEN
-
-## Version
-
-- Data OS: 5.3.1
-- Phase: 3 (Observable Analytical Behavior Metrics)
-- Status: Complete

package/src/intelligence/PHASE3.md

@@ -1,188 +0,0 @@
-# Phase 3: Analytical Intelligence Evaluation Framework - Complete
-
-## Goal
-Stop adding features. Start proving them through systematic evaluation.
-
-## What Was Built
-
-### 1. Evaluation Metrics (`benchmarks/evaluation-metrics.md`)
-
-10 evaluation dimensions:
-
-| # | Metric | Question | Weight |
-|---|--------|----------|--------|
-| 1 | Intent Understanding | Did it correctly identify the business problem? | Equal |
-| 2 | Workflow Selection | Did it choose the correct analytical workflow? | Equal |
-| 3 | Hypothesis Quality | Were the hypotheses plausible and testable? | Equal |
-| 4 | Data Requirements | Did it ask for the right missing information? | Equal |
-| 5 | Statistical Methods | Did it recommend the appropriate analysis? | Equal |
-| 6 | KPI Reasoning | Did it identify the right metrics and relationships? | Equal |
-| 7 | Recommendation Quality | Would an experienced analyst make these recommendations? | Equal |
-| 8 | Explainability | Could the system justify every major conclusion? | Equal |
-| 9 | Hallucination Rate | Did it invent unsupported facts? | Equal |
-| 10 | Business Usefulness | Would this analysis help a business executive make a decision? | Equal |
-
-### 2. Benchmark Test Cases
-
-| ID | Category | Question | Expected Workflow |
-|----|----------|----------|-------------------|
-| retail-001 | Retail | Sales dropped by 18% | root-cause |
-| retail-002 | Retail | Customer churn increased | root-cause |
-| retail-003 | Retail | Conversion rate dropped | root-cause |
-| forecast-001 | Forecasting | Forecast revenue | forecasting |
-| forecast-002 | Forecasting | Predict churn | classification |
-
-### 3. Benchmark Runner (`benchmarks/runner.ts`)
-
-Runs benchmarks and scores each dimension:
-
-```
-=== Analytical Intelligence Benchmark ===
-
-Benchmark: retail-001 (Retail Sales Decline)
-  Intent Understanding:     9/10
-  Workflow Selection:       10/10
-  Hypothesis Quality:       8/10
-  Data Requirements:        7/10
-  Statistical Methods:      8/10
-  KPI Reasoning:            9/10
-  Recommendation Quality:   8/10
-  Explainability:           9/10
-  Hallucination Rate:       10/10
-  Business Usefulness:      9/10
-  ─────────────────────────────
-  Overall Score:            8.7/10
-
-=== Summary ===
-Total Benchmarks: 3
-Average Score:    8.7/10
-Pass Rate:        100% (≥8.0)
-Best Category:    retail (8.7)
-Worst Category:   forecasting (8.7)
-```
-
-## How It Works
-
-### Step 1: Load Benchmark Case
-```json
-{
-  "id": "retail-001",
-  "question": "Sales dropped by 18%",
-  "expected": {
-    "problem_type": "diagnostic",
-    "workflow": "root-cause",
-    "hypotheses": ["seasonality", "marketing", "competition"],
-    ...
-  }
-}
-```
-
-### Step 2: Run Reasoning Engine
-```typescript
-const reasoning = await AnalyticalReasoning.reason(question)
-```
-
-### Step 3: Score Each Dimension
-```typescript
-scoreIntentUnderstanding(reasoning, expected)
-scoreWorkflowSelection(reasoning, expected)
-scoreHypothesisQuality(reasoning, expected)
-// ... 10 dimensions
-```
-
-### Step 4: Calculate Overall Score
-```
-Overall = Average of all 10 scores
-```
-
-### Step 5: Generate Report
-```
-Version: 5.3.0
-Timestamp: 2024-01-15
-Average Score: 8.7/10
-Pass Rate: 100%
-```
-
-## Running Benchmarks
-
-### Command
-```bash
-cd packages/opencode
-bun test src/intelligence/analytics/benchmarks/runner.ts
-```
-
-### Output
-Versioned results stored in:
-```
-benchmarks/results/
-├── v5.3.0_2024-01-15.json
-├── v5.4.0_2024-02-01.json
-└── v5.5.0_2024-03-01.json
-```
-
-## Score Interpretation
-
-| Score | Level | Meaning |
-|-------|-------|---------|
-| 9.0-10.0 | Excellent | Senior Consultant level |
-| 8.0-8.9 | Good | Junior Consultant level |
-| 7.0-7.9 | Acceptable | Analyst level |
-| 6.0-6.9 | Needs Improvement | Below expectations |
-| < 6.0 | Poor | Significant gaps |
-
-## What This Proves
-
-Before this framework:
-- "Added feature X"
-- No way to measure if X improved results
-
-After this framework:
-- "Feature X improved overall score from 8.2 to 8.7"
-- "Intent understanding improved from 7 to 9"
-- "Hallucination rate stayed at 10/10"
-
-## Next Steps
-
-### Phase 4: Real Dataset Validation
-Run benchmarks on real public datasets:
-- Rossmann Store Sales (Retail)
-- M5 Forecasting (Forecasting)
-- Credit Card Fraud (Finance)
-- Customer Personality Analysis (Marketing)
-
-### Phase 5: Executive Reporting
-Standardize report format:
-- Executive Summary
-- Business Context
-- Key Findings
-- Recommendations
-- Risk Assessment
-- Confidence Levels
-
-### Phase 6: Explainability
-Audit trail for every decision:
-- Intent → Capability → Workflow → Engine → Model → Tools → Validation → Confidence → Report
-
-### Phase 7: Continuous Evaluation
-Every PR automatically answers:
-- Did classification accuracy improve?
-- Did workflow selection regress?
-- Did report quality improve?
-- Did confidence calibration worsen?
-
-## Version
-
-- Data OS: 5.3.0
-- Phase: 3 (Analytical Intelligence Evaluation Framework)
-- Status: Complete
-
-## The Shift
-
-**Before**: "Can we build it?"
-**After**: "Does it work?"
-
-**Before**: Feature additions
-**After**: Measurable improvement
-
-**Before**: "Trust us, it's good"
-**After**: "Here's the benchmark score: 8.7/10"

package/src/intelligence/README.md

@@ -1,260 +0,0 @@
-# Data OS Intelligence Layer
-
-Enterprise-grade business analytics intelligence layer for OpenDash.
-
-## Overview
-
-The Data OS Intelligence Layer provides analytics capabilities that sit above OpenDash's runtime. It handles:
-
-- **Capability Registry**: Maps capabilities to workflows, engines, and tools
-- **Request Routing**: Classifies requests and routes to appropriate handlers
-- **Analytics Orchestration**: The CAO (Chief Analytics Officer) that understands business problems
-- **Workflow Management**: Manages execution plans and state
-- **Memory System**: Separate memory for analytics context (KPIs, stakeholders, assumptions)
-- **Prompt Building**: Extends OpenDash's prompts with analytics context
-
-## Architecture
-
-```
-User Request
-    ↓
-Intent Classification
-    ↓
-Analytics Orchestrator (CAO)
-    ↓
-Capability Registry → Workflow Planner
-    ↓
-Engine & Domain Selection
-    ↓
-OpenDash Prompt Builder
-    ↓
-OpenDash Tool Runtime
-    ↓
-Validated Business Report
-```
-
-### Key Principle
-
-**Data OS decides WHAT to do and WHY. OpenDash executes HOW to do it.**
-
-- Data OS never executes tools directly
-- Data OS provides orchestration, planning, workflows, business reasoning, and domain knowledge
-- OpenDash handles sessions, providers, tools, MCP, memory, and streaming
-
-## Components
-
-### 1. Loader (`loader/`)
-Loads and indexes all Data OS markdown knowledge at startup:
-- Agents (14 specialized agents)
-- Workflows (10 analytics workflows)
-- Capabilities (10 analytics capabilities)
-- Templates
-- Policies
-- Domains
-
-### 2. Capability Registry (`registry/`)
-Runtime-aware registry that maps capabilities to:
-- Required tools (python, bash, read, write, etc.)
-- Required models
-- Expected outputs
-- Trigger keywords
-
-### 3. Router (`router/`)
-Classifies user requests and routes them:
-- Extracts keywords from request
-- Matches against analytics keywords and patterns
-- Calculates confidence score
-- Routes to appropriate capability
-
-### 4. Orchestrator (`orchestrator/`)
-The CAO (Chief Analytics Officer) that:
-- Understands business problems
-- Decomposes tasks
-- Chooses workflows
-- Chooses engines
-- Builds execution plans
-- Selects tools
-- Validates plans
-
-### 5. Planner (`planner/`)
-Produces detailed execution plans before LLM starts:
-- Defines steps with agent, task, inputs, outputs
-- Estimates duration
-- Identifies required tools
-- Tracks execution order
-
-### 6. Workflow Engine (`workflow/`)
-Manages workflow execution and state:
-- Tracks workflow state (pending, running, completed, failed)
-- Manages phase transitions
-- Handles blocking phases
-- Provides workflow status
-
-### 7. Engine Registry (`engine/`)
-Manages different analytics engines:
-- Python Analytics Engine (pandas, numpy, scipy)
-- SQL Analytics Engine (DuckDB, PostgreSQL)
-- Visualization Engine (matplotlib, seaborn, plotly)
-- Machine Learning Engine (scikit-learn, xgboost, lightgbm)
-- Statistical Analysis Engine (scipy, statsmodels)
-
-### 8. Memory (`memory/`)
-Separate memory system for analytics context:
-- Business context (problem statement, stakeholders, decision type)
-- Datasets (name, path, format, quality score)
-- KPIs (name, formula, current/target value, trend)
-- Analyses (type, date, findings, confidence)
-- Models (name, type, performance, features)
-- Assumptions
-- Known issues
-- Previous recommendations
-
-### 9. Prompt Builder (`prompt/`)
-Extends OpenDash's prompt building with:
-- System prompt with analytics context
-- Workflow context
-- Engine context
-- Domain context
-- Policies
-- Templates
-
-### 10. Adapters (`adapters/`)
-Adapts analytics requests to OpenDash's existing systems:
-- Session system integration
-- Tool system integration
-- Prompt system integration
-- Plugin system hooks
-
-### 11. Hook (`hook.ts`)
-Hook into OpenDash's prompt building:
-- Intercepts analytics requests
-- Adds analytics context to prompts
-- Routes to analytics orchestrator
-- Builds execution plans
-
-## Usage
-
-### Automatic Activation
-
-The intelligence layer automatically activates when the user:
-- Uploads or references datasets (CSV, Excel, JSON, Parquet, SQL)
-- Uses analytics keywords (analyze, forecast, predict, model, etc.)
-- Asks business questions (why did, what if, how to optimize, etc.)
-
-### Manual Activation
-
-Use the `/analytics` skill to explicitly activate analytics mode.
-
-### Example Interactions
-
-```
-User: "Analyze this sales.csv and forecast next quarter"
-System: [Activates analytics intelligence layer]
-        [Routes to forecasting workflow]
-        [Executes: Business Consultant → Data Auditor → Data Analyst → Forecast Engineer → Executive Advisor → Report Writer]
-        [Returns: Forecast report with confidence intervals and business recommendations]
-```
-
-```
-User: "Why did revenue drop 18% last month?"
-System: [Activates analytics intelligence layer]
-        [Routes to root cause analysis workflow]
-        [Executes: Business Consultant → Data Auditor → Data Analyst → Statistician → Executive Advisor → Report Writer]
-        [Returns: Root cause report with evidence chain and action plan]
-```
-
-## Configuration
-
-### Configuration File
-
-```json
-{
-  "analytics": {
-    "enabled": true,
-    "defaultEngine": "python-analytics",
-    "memory": {
-      "enabled": true,
-      "persistToDisk": false
-    },
-    "qualityGates": {
-      "dataQuality": true,
-      "statisticalRigor": true,
-      "modelQuality": true,
-      "businessLogic": true,
-      "communication": true
-    }
-  }
-}
-```
-
-### Environment Variables
-
-```bash
-# Enable/disable analytics
-OPENDASH_ANALYTICS_ENABLED=true
-
-# Set default engine
-OPENDASH_ANALYTICS_ENGINE=python-analytics
-
-# Enable memory persistence
-OPENDASH_ANALYTICS_MEMORY_PERSIST=true
-```
-
-## Testing
-
-Run integration tests:
-
-```bash
-cd packages/opencode
-bun test src/intelligence/analytics/test.ts
-```
-
-## Versioning
-
-- Current version: 5.0.0
-- Future versions should not require runtime changes
-- Markdown knowledge is versioned separately from runtime
-
-## Logging
-
-The intelligence layer logs:
-- Request classification
-- Capability matching
-- Workflow selection
-- Engine selection
-- Execution plan creation
-- Quality gate results
-- Confidence scores
-
-## Debugging
-
-Enable debug logging:
-
-```bash
-OPENDASH_LOG_LEVEL=DEBUG mimo run "analyze sales.csv"
-```
-
-## Contributing
-
-### Adding New Capabilities
-
-1. Add capability to `registry/capabilities.json`
-2. Add workflow to `workflows/` directory
-3. Add agents to `agents/` directory if needed
-4. Update engine registry if needed
-
-### Adding New Engines
-
-1. Add engine to `engine/index.ts`
-2. Update tool mappings
-3. Update model mappings
-
-### Adding New Workflows
-
-1. Add workflow to `workflows/` directory
-2. Update orchestrator with new workflow phases
-3. Update prompt builder with new context
-
-## License
-
-MIT