@memberjunction/db-auto-doc 2.116.0 → 2.118.0

package/README.md

@@ -1,244 +1,731 @@
-# Database Auto-Documentation Generator
+# DBAutoDoc - AI-Powered Database Documentation Generator
 
-AI-powered documentation generator for SQL Server databases. Analyzes your database structure, uses AI to generate comprehensive table and column descriptions, and saves them as SQL Server extended properties.
-
-## 🚀 **Standalone Tool - No MemberJunction Runtime Required**
-
-This tool works with **ANY** SQL Server database. You don't need MemberJunction installed or running.
+Automatically generate comprehensive documentation for SQL Server, MySQL, and PostgreSQL databases using AI. DBAutoDoc analyzes your database structure, uses Large Language Models to understand the purpose of tables and columns, and saves descriptions as database metadata (extended properties for SQL Server, comments for MySQL/PostgreSQL).
 
 ## Features
 
-- **🤖 AI-Powered**: Uses LLMs (OpenAI, Anthropic, etc.) to generate intelligent descriptions
-- **🔄 Human-in-Loop**: Interactive mode to provide context and approve AI-generated descriptions
-- **💾 State Management**: JSON state file tracks progress, user input, and AI generations across runs
-- **🎯 Incremental**: Only processes new or changed tables on subsequent runs
-- **🔍 Smart Analysis**:
-  - Dependency graph analysis (documents root tables first)
-  - Pattern detection (lookup tables, bridge tables, audit tables)
-  - Data profiling (sample data, statistics, pattern recognition)
-  - Constraint analysis (PKs, FKs, CHECK, UNIQUE)
-- **📊 Multiple Outputs**:
-  - SQL scripts with `sp_addextendedproperty` statements
-  - Markdown documentation
-  - Updated state file for next run
+### Core Capabilities
+- **🤖 AI-Powered Analysis** - Uses OpenAI, Anthropic, Google, or Groq to generate intelligent descriptions
+- **🔄 Iterative Refinement** - Multi-pass analysis with backpropagation for accuracy
+- **📊 Topological Processing** - Analyzes tables in dependency order for better context
+- **📈 Data-Driven** - Leverages cardinality, statistics, and sample data for insights
+- **🎯 Convergence Detection** - Automatically knows when analysis is complete
+- **💾 State Tracking** - Full audit trail of all iterations and reasoning
+- **🔌 Standalone** - Works with ANY database, no MemberJunction required
+
+### Multi-Database Support
+- **SQL Server** - Full support with extended properties
+- **PostgreSQL** - Complete implementation with COMMENT syntax
+- **MySQL** - Full support with column/table comments
+- **Unified Interface** - Single configuration approach across all databases
+
+### Advanced Features
+- **🔍 Relationship Discovery** - Automatically detect missing primary and foreign keys using statistical analysis and LLM validation
+- **🛡️ Granular Guardrails** - Multi-level resource controls (run, phase, iteration limits)
+- **⏸️ Resume Capability** - Pause and resume analysis from checkpoint state files
+- **📦 Programmatic API** - Use as a library in your own applications
+- **🔧 Extensible** - Custom database drivers and analysis plugins
+
+### Output Formats
+- **SQL Scripts** - Database-specific metadata scripts (extended properties, comments)
+- **Markdown Documentation** - Human-readable docs with ERD diagrams
+- **HTML Documentation** - Interactive, searchable documentation with embedded CSS/JS
+- **CSV Exports** - Spreadsheet-ready table and column data
+- **Mermaid Diagrams** - Standalone ERD files (.mmd and .html)
+- **Analysis Reports** - Detailed metrics and quality assessments
 
 ## Installation
 
+### Global Installation (Recommended for DBAs)
+
 ```bash
-# Install globally (for standalone use)
 npm install -g @memberjunction/db-auto-doc
+```
 
-# Or use with npx
-npx @memberjunction/db-auto-doc
+### Within MemberJunction Project
 
-# Or use via MJ CLI (if you have MemberJunction installed)
-mj dbdoc --help
+```bash
+npm install @memberjunction/db-auto-doc
+```
+
+### As a Library Dependency
+
+```bash
+npm install @memberjunction/db-auto-doc --save
 ```
 
 ## Quick Start
 
-### Standalone CLI
+### 1. Initialize
 
 ```bash
-# 1. Initialize project
 db-auto-doc init
+```
+
+This interactive wizard will:
+- Configure database connection
+- Set up AI provider (OpenAI, Anthropic, Google, or Groq)
+- Configure guardrails and resource limits
+- Optionally add seed context for better analysis
+- Create `config.json`
 
-# 2. Edit .env and add your AI API key
-# AI_API_KEY=sk-your-key-here
+### 2. Analyze
 
-# 3. Analyze database
+```bash
 db-auto-doc analyze
+```
+
+This will:
+- Introspect your database structure
+- Analyze data (cardinality, statistics, patterns)
+- Optionally discover missing primary and foreign keys
+- Build dependency graph
+- Run iterative AI analysis with backpropagation
+- Perform sanity checks
+- Save state to `db-doc-state.json`
 
-# 4. Review results
-db-auto-doc review
+### 3. Export
 
-# 5. Export documentation
-db-auto-doc export --format=all
+```bash
+db-auto-doc export --sql --markdown --html --csv --mermaid
 ```
 
-### Via MJ CLI (MemberJunction Users)
+This generates:
+- **SQL Script**: Database-specific metadata statements
+- **Markdown Documentation**: Human-readable docs with ERD links
+- **HTML Documentation**: Interactive searchable documentation
+- **CSV Files**: tables.csv and columns.csv for spreadsheet analysis
+- **Mermaid Diagrams**: erd.mmd and erd.html for visualization
+
+Optionally apply directly to database:
 
 ```bash
-# Same commands, different prefix
-mj dbdoc init
-mj dbdoc analyze --schemas=dbo
-mj dbdoc review --unapproved-only
-mj dbdoc export --approved-only --execute
+db-auto-doc export --sql --apply
 ```
 
-### Programmatic Usage
+### 4. Check Status
 
-```typescript
-import {
-  DatabaseConnection,
-  StateManager,
-  DatabaseAnalyzer,
-  SimpleAIClient,
-} from '@memberjunction/db-auto-doc';
+```bash
+db-auto-doc status
+```
 
-// Initialize
-const connection = DatabaseConnection.fromEnv();
-const stateManager = new StateManager();
-const aiClient = new SimpleAIClient();
-const analyzer = new DatabaseAnalyzer(connection, stateManager, aiClient);
+Shows:
+- Analysis progress and phase completion
+- Convergence status
+- Low-confidence tables and columns
+- Token usage, cost, and duration
+- Guardrail status and warnings
 
-// Analyze
-await analyzer.analyze({ schemas: ['dbo'] });
+### 5. Resume Analysis
 
-// Export
-import { SQLGenerator, MarkdownGenerator } from '@memberjunction/db-auto-doc';
-const state = stateManager.getState();
-const sqlGen = new SQLGenerator();
-const sql = sqlGen.generate(state);
+```bash
+db-auto-doc analyze --resume ./db-doc-state.json
 ```
 
-## CLI Commands
+Resume a previous analysis from a checkpoint state file, useful for:
+- Continuing after hitting guardrail limits
+- Recovering from interruptions
+- Incremental database updates
+
+## How It Works
+
+### Topological Analysis
 
-### `db-auto-doc init`
-Initialize new documentation project
-- Prompts for database connection
-- Creates `.env` file
-- Creates `db-doc-state.json`
-- Optionally asks seed questions
+DBAutoDoc processes tables in dependency order:
 
-### `db-auto-doc analyze`
-Analyze database and generate documentation
-- `--interactive` - Ask questions during analysis
-- `--incremental` - Only process new/changed tables
-- `--schemas <schemas>` - Comma-separated schema list
-- `--batch` - Non-interactive mode
+```
+Level 0: Users, Products, Categories (no dependencies)
+  ↓
+Level 1: Orders (depends on Users), ProductCategories (Products + Categories)
+  ↓
+Level 2: OrderItems (depends on Orders + Products)
+  ↓
+Level 3: Shipments (depends on OrderItems)
+```
 
-### `db-auto-doc review`
-Review and approve AI-generated documentation
-- `--schema <schema>` - Review specific schema
-- `--unapproved-only` - Only show unapproved items
+Processing in this order allows child tables to benefit from parent table context.
 
-### `db-auto-doc export`
-Generate output files
-- `--format <format>` - sql|markdown|all (default: all)
-- `--output <path>` - Output directory
-- `--execute` - Execute SQL script (apply to database)
-- `--approved-only` - Only export approved items
+### Relationship Discovery
 
-### `db-auto-doc reset`
-Reset state file
-- `--all` - Reset entire state file
+For legacy databases missing primary/foreign key constraints, DBAutoDoc can:
+- **Detect Primary Keys** using statistical analysis (uniqueness, nullability, cardinality)
+- **Find Foreign Keys** using value overlap analysis and naming patterns
+- **LLM Validation** to verify discovered relationships make business sense
+- **Backpropagation** to refine parent table analysis based on child relationships
 
-## Configuration
+Triggered automatically when:
+- Tables lack primary key constraints
+- Insufficient foreign key relationships detected (below threshold)
 
-Create a `.env` file:
+### Backpropagation
 
-```env
-# Database Connection
-DB_SERVER=localhost
-DB_DATABASE=YourDatabase
-DB_USER=sa
-DB_PASSWORD=YourPassword
-DB_ENCRYPT=true
-DB_TRUST_SERVER_CERTIFICATE=true
+After analyzing child tables, DBAutoDoc can detect insights about parent tables and trigger re-analysis:
 
-# AI Configuration
-AI_PROVIDER=openai
-AI_MODEL=gpt-4
-AI_API_KEY=sk-your-api-key-here
+```
+Level 0: "Persons" → Initially thinks: "General contact information"
+  ↓
+Level 1: "Students" table reveals Persons.Type has values: Student, Teacher, Staff
+  ↓
+BACKPROPAGATE to Level 0: "Persons" → Revise to: "School personnel with role-based typing"
 ```
 
-## State File
+### Convergence
+
+Analysis stops when:
+1. **No changes** in last N iterations (stability window)
+2. **All tables** meet confidence threshold
+3. **Max iterations** reached
+4. **Guardrail limits** exceeded (tokens, cost, duration)
+
+### Granular Guardrails
+
+Multi-level resource controls ensure analysis stays within bounds:
 
-The `db-doc-state.json` file tracks everything:
+**Run-Level Limits**:
+- `maxTokensPerRun`: Total token budget for entire analysis
+- `maxDurationSeconds`: Maximum wall-clock time
+- `maxCostDollars`: Maximum AI cost
+
+**Phase-Level Limits**:
+- `maxTokensPerPhase.discovery`: Budget for relationship discovery
+- `maxTokensPerPhase.analysis`: Budget for description generation
+- `maxTokensPerPhase.sanityChecks`: Budget for validation
+
+**Iteration-Level Limits**:
+- `maxTokensPerIteration`: Per-iteration token cap
+- `maxIterationDurationSeconds`: Per-iteration time limit
+
+**Warning Thresholds**:
+- Configurable percentage-based warnings (default 80-85%)
+- Early notification before hitting hard limits
+
+### Data Analysis
+
+For each column, DBAutoDoc collects:
+- **Cardinality**: Distinct value counts
+- **Statistics**: Min, max, average, standard deviation
+- **Patterns**: Common prefixes, format detection
+- **Value Distribution**: Actual enum values if low cardinality
+- **Sample Data**: Stratified sampling across value ranges
+
+This rich context enables AI to make accurate inferences.
+
+## Configuration
+
+### SQL Server Configuration
 
 ```json
 {
-  "version": "1.0",
+  "version": "1.0.0",
   "database": {
-    "server": "localhost",
-    "database": "MyDatabase"
+    "provider": "sqlserver",
+    "host": "localhost",
+    "database": "MyDatabase",
+    "user": "sa",
+    "password": "YourPassword",
+    "encrypt": true,
+    "trustServerCertificate": false
   },
-  "seedContext": {
-    "overallPurpose": "E-commerce platform",
-    "businessDomains": ["Sales", "Inventory"]
+  "ai": {
+    "provider": "openai",
+    "model": "gpt-4-turbo-preview",
+    "apiKey": "sk-...",
+    "temperature": 0.1,
+    "maxTokens": 8000,
+    "effortLevel": 50
   },
-  "schemas": {
-    "dbo": {
-      "tables": {
-        "Customers": {
-          "userNotes": "Merged from Stripe and internal CRM",
-          "userApproved": true,
-          "aiGenerated": {
-            "description": "Primary customer records...",
-            "confidence": 0.85
-          },
-          "finalDescription": "...",
-          "columns": { }
-        }
+  "analysis": {
+    "cardinalityThreshold": 20,
+    "sampleSize": 10,
+    "includeStatistics": true,
+    "includePatternAnalysis": true,
+    "convergence": {
+      "maxIterations": 50,
+      "stabilityWindow": 2,
+      "confidenceThreshold": 0.85
+    },
+    "backpropagation": {
+      "enabled": true,
+      "maxDepth": 3
+    },
+    "sanityChecks": {
+      "dependencyLevel": true,
+      "schemaLevel": true,
+      "crossSchema": true
+    },
+    "guardrails": {
+      "enabled": true,
+      "stopOnExceeded": true,
+      "maxTokensPerRun": 250000,
+      "maxDurationSeconds": 3600,
+      "maxCostDollars": 50,
+      "maxTokensPerPhase": {
+        "discovery": 100000,
+        "analysis": 150000,
+        "sanityChecks": 50000
+      },
+      "maxTokensPerIteration": 50000,
+      "maxIterationDurationSeconds": 600,
+      "warnThresholds": {
+        "tokenPercentage": 80,
+        "durationPercentage": 80,
+        "costPercentage": 80,
+        "iterationTokenPercentage": 85,
+        "phaseTokenPercentage": 85
+      }
+    },
+    "relationshipDiscovery": {
+      "enabled": true,
+      "triggers": {
+        "runOnMissingPKs": true,
+        "runOnInsufficientFKs": true,
+        "fkDeficitThreshold": 0.4
+      },
+      "tokenBudget": {
+        "ratioOfTotal": 0.4
+      },
+      "confidence": {
+        "primaryKeyMinimum": 0.7,
+        "foreignKeyMinimum": 0.6,
+        "llmValidationThreshold": 0.8
+      },
+      "sampling": {
+        "maxRowsPerTable": 1000,
+        "valueOverlapSampleSize": 100,
+        "statisticalSignificance": 100,
+        "compositeKeyMaxColumns": 3
+      },
+      "patterns": {
+        "primaryKeyNames": ["^id$", ".*_id$", "^pk_.*", ".*_key$"],
+        "foreignKeyNames": [".*_id$", ".*_fk$", "^fk_.*"]
+      },
+      "llmValidation": {
+        "enabled": true,
+        "batchSize": 10
+      },
+      "backpropagation": {
+        "enabled": true,
+        "maxIterations": 5
       }
     }
   },
-  "runHistory": [ ]
+  "output": {
+    "stateFile": "./db-doc-state.json",
+    "outputDir": "./output",
+    "sqlFile": "./output/add-descriptions.sql",
+    "markdownFile": "./output/database-documentation.md"
+  },
+  "schemas": {
+    "exclude": ["sys", "INFORMATION_SCHEMA"]
+  },
+  "tables": {
+    "exclude": ["sysdiagrams", "__MigrationHistory"]
+  }
 }
 ```
 
-## Example Workflow
+### PostgreSQL Configuration
 
-```bash
-# First run
-db-auto-doc init
-db-auto-doc analyze --interactive
-db-auto-doc review
-db-auto-doc export --format=all
+```json
+{
+  "version": "1.0.0",
+  "database": {
+    "provider": "postgresql",
+    "host": "localhost",
+    "port": 5432,
+    "database": "mydatabase",
+    "user": "postgres",
+    "password": "YourPassword",
+    "ssl": false
+  },
+  "ai": {
+    "provider": "openai",
+    "model": "gpt-4-turbo-preview",
+    "apiKey": "sk-...",
+    "temperature": 0.1,
+    "maxTokens": 8000
+  },
+  "analysis": {
+    "cardinalityThreshold": 20,
+    "sampleSize": 10,
+    "includeStatistics": true,
+    "guardrails": {
+      "enabled": true,
+      "maxTokensPerRun": 250000
+    }
+  },
+  "output": {
+    "stateFile": "./db-doc-state.json",
+    "outputDir": "./output",
+    "sqlFile": "./output/add-descriptions.sql",
+    "markdownFile": "./output/database-documentation.md"
+  },
+  "schemas": {
+    "exclude": ["pg_catalog", "information_schema"]
+  }
+}
+```
+
+### MySQL Configuration
+
+```json
+{
+  "version": "1.0.0",
+  "database": {
+    "provider": "mysql",
+    "host": "localhost",
+    "port": 3306,
+    "database": "mydatabase",
+    "user": "root",
+    "password": "YourPassword"
+  },
+  "ai": {
+    "provider": "openai",
+    "model": "gpt-4-turbo-preview",
+    "apiKey": "sk-...",
+    "temperature": 0.1,
+    "maxTokens": 8000
+  },
+  "analysis": {
+    "cardinalityThreshold": 20,
+    "sampleSize": 10,
+    "includeStatistics": true,
+    "guardrails": {
+      "enabled": true,
+      "maxTokensPerRun": 250000
+    }
+  },
+  "output": {
+    "stateFile": "./db-doc-state.json",
+    "outputDir": "./output",
+    "sqlFile": "./output/add-descriptions.sql",
+    "markdownFile": "./output/database-documentation.md"
+  },
+  "schemas": {
+    "exclude": ["mysql", "information_schema", "performance_schema", "sys"]
+  }
+}
+```
 
-# Add context and refine
-# Edit db-doc-state.json to add notes
-db-auto-doc analyze --incremental
-db-auto-doc review --unapproved-only
+## Supported AI Providers
+
+DBAutoDoc integrates with MemberJunction's AI provider system, supporting:
+
+### OpenAI
+```json
+{
+  "provider": "OpenAILLM",
+  "model": "gpt-4-turbo-preview",
+  "apiKey": "sk-..."
+}
+```
 
-# Ready for production
-db-auto-doc export --approved-only --execute
+### Anthropic
+```json
+{
+  "provider": "AnthropicLLM",
+  "model": "claude-3-5-sonnet-20241022",
+  "apiKey": "sk-ant-..."
+}
 ```
 
-## How It Works
+### Google
+```json
+{
+  "provider": "GoogleLLM",
+  "model": "gemini-1.5-pro",
+  "apiKey": "..."
+}
+```
+
+### Groq
+```json
+{
+  "provider": "GroqLLM",
+  "model": "llama-3.3-70b-versatile",
+  "apiKey": "gsk_..."
+}
+```
+
+### Other Providers
+Any BaseLLM-compatible provider registered with MemberJunction can be used.
+
+## State File
+
+The `db-doc-state.json` file tracks:
+- All schemas, tables, and columns
+- **Description iterations** with reasoning and confidence
+- **Analysis runs** with metrics (tokens, cost, duration)
+- **Processing logs** for debugging
+- **Relationship discovery results** (primary keys, foreign keys)
+- **Guardrail metrics** (phase and iteration budgets)
+
+### Iteration Tracking
+
+Each description has an iteration history:
+
+```json
+{
+  "descriptionIterations": [
+    {
+      "description": "Initial hypothesis...",
+      "reasoning": "Based on column names...",
+      "generatedAt": "2024-01-15T10:00:00Z",
+      "modelUsed": "gpt-4",
+      "confidence": 0.75,
+      "triggeredBy": "initial"
+    },
+    {
+      "description": "Revised hypothesis...",
+      "reasoning": "Child table analysis revealed...",
+      "generatedAt": "2024-01-15T10:05:00Z",
+      "modelUsed": "gpt-4",
+      "confidence": 0.92,
+      "triggeredBy": "backpropagation",
+      "changedFrom": "Initial hypothesis..."
+    }
+  ]
+}
+```
 
-1. **Introspection**: Queries SQL Server system catalogs
-2. **Profiling**: Samples data and analyzes patterns
-3. **AI Generation**: Sends context to LLM for descriptions
-4. **Human Review**: User approves/refines results
-5. **Output**: Generates SQL scripts and markdown docs
-6. **Application**: Optionally executes SQL to add extended properties
+## Programmatic Usage
+
+DBAutoDoc can be used as a library with a comprehensive programmatic API:
+
+### Simple API (Recommended)
+
+```typescript
+import { DBAutoDocAPI } from '@memberjunction/db-auto-doc';
+
+const api = new DBAutoDocAPI();
+
+// Analyze database
+const result = await api.analyze({
+  database: {
+    provider: 'sqlserver',
+    host: 'localhost',
+    database: 'MyDB',
+    user: 'sa',
+    password: 'password'
+  },
+  ai: {
+    provider: 'OpenAILLM',
+    model: 'gpt-4-turbo-preview',
+    apiKey: 'sk-...'
+  },
+  analysis: {
+    convergence: { maxIterations: 10 },
+    guardrails: { maxTokensPerRun: 100000 }
+  },
+  output: {
+    outputDir: './output'
+  },
+  onProgress: (message, data) => {
+    console.log(message, data);
+  }
+});
+
+// Resume from state file
+const resumed = await api.resume('./db-doc-state.json', {
+  analysis: {
+    convergence: { maxIterations: 20 }
+  }
+});
+
+// Export documentation
+const exported = await api.export('./db-doc-state.json', {
+  formats: ['sql', 'markdown', 'html', 'csv', 'mermaid'],
+  outputDir: './docs',
+  applyToDatabase: true
+});
+
+// Get analysis status
+const status = await api.getStatus('./db-doc-state.json');
+console.log('Progress:', status.progress);
+console.log('Tokens used:', status.metrics.totalTokens);
+console.log('Estimated cost:', status.metrics.estimatedCost);
+```
+
+### Advanced API (Full Control)
+
+```typescript
+import {
+  ConfigLoader,
+  DatabaseConnection,
+  Introspector,
+  TopologicalSorter,
+  StateManager,
+  PromptEngine,
+  AnalysisEngine,
+  GuardrailsManager,
+  SQLGenerator,
+  MarkdownGenerator,
+  HTMLGenerator,
+  CSVGenerator,
+  MermaidGenerator
+} from '@memberjunction/db-auto-doc';
+
+// Load config
+const config = await ConfigLoader.load('./config.json');
+
+// Connect to database
+const db = new DatabaseConnection(config.database);
+await db.connect();
+
+// Introspect
+const driver = db.getDriver();
+const introspector = new Introspector(driver);
+const schemas = await introspector.getSchemas(config.schemas, config.tables);
+
+// Initialize analysis components
+const promptEngine = new PromptEngine(config.ai, './prompts');
+await promptEngine.initialize();
+
+const stateManager = new StateManager(config.output.stateFile);
+const state = stateManager.createInitialState(config.database.database, config.database.server);
+state.schemas = schemas;
+
+const guardrails = new GuardrailsManager(config.analysis.guardrails);
+const iterationTracker = new IterationTracker();
+
+// Run analysis
+const analysisEngine = new AnalysisEngine(config, promptEngine, stateManager, iterationTracker);
+// ... custom analysis workflow
+
+// Generate outputs
+const sqlGen = new SQLGenerator();
+const sql = sqlGen.generate(state, { approvedOnly: false });
+
+const mdGen = new MarkdownGenerator();
+const markdown = mdGen.generate(state);
+
+const htmlGen = new HTMLGenerator();
+const html = htmlGen.generate(state, { confidenceThreshold: 0.7 });
+
+const csvGen = new CSVGenerator();
+const { tables, columns } = csvGen.generate(state);
+
+const mermaidGen = new MermaidGenerator();
+const erdDiagram = mermaidGen.generate(state);
+const erdHtml = mermaidGen.generateHtml(state);
+```
+
+## Cost Estimation
+
+Typical costs (will vary by database size and complexity):
+
+| Database Size | Tables | Iterations | Tokens | Cost (GPT-4) | Cost (Groq) |
+|---------------|--------|------------|--------|--------------|-------------|
+| Small | 10-20 | 2-3 | ~50K | $0.50 | $0.02 |
+| Medium | 50-100 | 3-5 | ~200K | $2.00 | $0.08 |
+| Large | 200+ | 5-8 | ~500K | $5.00 | $0.20 |
+| Enterprise | 500+ | 8-15 | ~1.5M | $15.00 | $0.60 |
+
+**With Relationship Discovery**: Add 25-40% to token/cost estimates for databases with missing constraints.
+
+**Guardrails** help control costs by setting hard limits on token usage and runtime.
+
+## Best Practices
+
+1. **Start with guardrails** - Set reasonable token/cost limits to avoid surprises
+2. **Add seed context** - Helps AI understand database purpose and domain
+3. **Review low-confidence items** - Focus manual effort where AI is uncertain
+4. **Use backpropagation** - Improves accuracy significantly
+5. **Enable relationship discovery** - For legacy databases missing constraints
+6. **Filter exports** - Use `--confidence-threshold` to only apply high-confidence descriptions
+7. **Iterate** - Run analysis multiple times if first pass isn't satisfactory
+8. **Resume from checkpoints** - Save costs by continuing previous runs
+9. **Use appropriate models** - Balance cost vs. quality (GPT-4 vs. Groq)
+10. **Export multiple formats** - HTML for browsing, CSV for analysis, SQL for database
+
+## Troubleshooting
+
+### "Connection failed"
+- Check server, database, user, password in config
+- Verify database server is running and accessible
+- Check firewall rules and network connectivity
+- For PostgreSQL: verify SSL settings
+- For MySQL: check port and authentication method
+
+### "Analysis not converging"
+- Increase `maxIterations` in config
+- Lower `confidenceThreshold`
+- Add more seed context
+- Check warnings in state file for specific issues
+- Review guardrail limits (may be hitting token budget)
+
+### "High token usage"
+- Enable guardrails with appropriate limits
+- Reduce `maxTokens` per prompt
+- Filter schemas/tables to focus on subset
+- Use cheaper model (Groq instead of GPT-4)
+- Disable relationship discovery if not needed
+
+### "Guardrail limits exceeded"
+- Review metrics in state file
+- Adjust limits upward if budget allows
+- Use `--resume` to continue from checkpoint
+- Focus on specific schemas/tables
+- Reduce iteration count
+
+### "Relationship discovery not finding keys"
+- Check confidence thresholds (may be too high)
+- Review statistical significance settings
+- Enable LLM validation for better accuracy
+- Check naming patterns configuration
+- Verify sample size is adequate
+
+## Documentation
+
+Comprehensive documentation is available in the `docs/` folder:
+
+- **[USER_GUIDE.md](./docs/USER_GUIDE.md)** - Complete user documentation
+- **[ARCHITECTURE.md](./docs/ARCHITECTURE.md)** - Technical architecture and design
+- **[API_USAGE.md](./docs/API_USAGE.md)** - Programmatic API examples
+- **[GUARDRAILS.md](./docs/GUARDRAILS.md)** - Guardrails system documentation
+- **[CHANGES.md](./docs/CHANGES.md)** - Recent changes and enhancements
 
 ## Architecture
 
-Built following the **AI CLI pattern** (similar to `@memberjunction/ai-cli`):
-- **oclif-based commands** in `src/commands/` (init, analyze, review, export, reset)
-- **Standalone package** with zero MJ runtime dependencies
-- **MJCLI integration** via thin delegation commands in `packages/MJCLI/src/commands/dbdoc/`
-- **Reusable services** exported for programmatic use
-- **State file architecture** enables incremental refinement across runs
+DBAutoDoc uses a sophisticated multi-phase architecture:
 
-## Future Enhancements
+1. **Discovery Phase** - Introspection and optional relationship discovery
+2. **Analysis Phase** - Iterative LLM-based description generation
+3. **Sanity Check Phase** - Validation and quality assurance
+4. **Export Phase** - Multi-format documentation generation
 
-Possible future additions (not needed for current functionality):
-- Support for more AI providers (Groq, Cerebras, local models)
-- PostgreSQL/MySQL versions
-- Web UI for review
-- Dependency graph visualization
-- CI/CD integration examples
-- Schema diff detection for automatic re-documentation
+See [ARCHITECTURE.md](./docs/ARCHITECTURE.md) for comprehensive architecture documentation, including:
+- Phase flow diagrams
+- Extension points for customization
+- Database driver development guide
+- LLM intelligence strategy
 
-## Requirements
+## Contributing
 
-- Node.js 18+
-- SQL Server database access
-- OpenAI or Anthropic API key
+DBAutoDoc is part of the MemberJunction project. Contributions welcome!
 
 ## License
 
-MIT License - see LICENSE file for details
+MIT
+
+## Demo Databases
+
+### LousyDB - Legacy Database Demo
+
+Located in `/Demos/LousyDB/`, this demo showcases **Relationship Discovery** capabilities on a realistic legacy database:
+
+- ❌ **Zero metadata** - No PK or FK constraints defined
+- 🔤 **Cryptic naming** - Short abbreviations (`cst`, `ord`, `pmt`)
+- 🔡 **Single-char codes** - Undocumented status values (`A`, `T`, `P`)
+- 💔 **Data quality issues** - Orphaned records, NULLs, duplicates
+- 📊 **20 tables** across 2 schemas with 1000+ rows
 
-## Support
+Perfect for testing DBAutoDoc's ability to **reverse-engineer** poorly-documented databases.
 
-- GitHub Issues: https://github.com/MemberJunction/MJ/issues
-- Documentation: https://docs.memberjunction.org
+See `/Demos/LousyDB/README.md` for details and testing instructions.
 
-## Credits
+## Links
 
-Built by the MemberJunction team for the SQL Server community.
+- **GitHub**: https://github.com/MemberJunction/MJ
+- **Documentation**: https://docs.memberjunction.org
+- **Support**: https://github.com/MemberJunction/MJ/issues