pgterra 0.1.0 → 0.1.2

package/README.md

@@ -1,544 +1,109 @@
-# PGTerra
+# pgterra
 
-A declarative Infrastructure as Code tool for PostgreSQL databases, similar to Terraform but specifically designed for database schema management.
+Declarative schema management for Postgres. Define your desired schema, pgterra handles the migrations.
 
-## 🔍 Overview
+## Quick Start
 
-PGTerra follows a declarative approach where you define your **desired database state** in a `schema.sql` file, and the tool automatically figures out what changes are needed to bring your database to that state. No more writing manual migration scripts - just describe what you want and let PGTerra handle the rest!
-
-### Key Concepts
-
-- **Declarative, Not Imperative**: You describe WHAT you want, not HOW to get there
-- **State-Based Management**: Compare current vs desired state and generate minimal changes
-- **Safety First**: Preview changes with `plan` before applying them
-- **Idempotent Operations**: Running the same schema multiple times is safe
-
-## 🏗️ How It Works
-
-```mermaid
-graph TD
-    A["schema.sql<br/>(Desired State)"] --> B["Schema Parser"]
-    C["PostgreSQL Database<br/>(Current State)"] --> D["Database Inspector"]
-
-    B --> E["Desired Schema<br/>(Parsed Objects)"]
-    D --> F["Current Schema<br/>(Queried Objects)"]
-
-    E --> G["Schema Differ"]
-    F --> G
-
-    G --> H["Migration Plan<br/>(SQL Statements)"]
-
-    H --> I{"Command Type"}
-    I -->|plan| J["Display Changes"]
-    I -->|apply| K["Execute Changes"]
-
-    K --> L["Updated Database"]
-```
-
-### Architecture Components
-
-#### 1. **Schema Parser**
-
-- Converts your `schema.sql` file into structured objects
-- Uses `sql-parser-cst` for robust SQL parsing
-- Handles CREATE TABLE statements with columns, constraints, and data types
-
-#### 2. **Database Inspector**
-
-- Queries the current database structure using PostgreSQL's `information_schema`
-- Extracts table definitions, column details, constraints, and metadata
-- Normalizes database state into the same format as parsed schema
-
-#### 3. **Schema Differ**
-
-- Compares desired vs current state
-- Generates optimized migration plan with proper operation ordering
-- Handles complex scenarios like type conversions, constraint changes, and data preservation
-
-#### 4. **Migration Planner & Executor**
-
-- Orchestrates the diffing process safely
-- Executes SQL statements with proper error handling
-- Provides detailed feedback on changes applied
-
-## 🔄 Step-by-Step Process
-
-```mermaid
-sequenceDiagram
-    participant CLI as CLI Command<br/>(plan/apply)
-    participant SS as Schema Service
-    participant SP as Schema Parser
-    participant DI as Database Inspector
-    participant SD as Schema Differ
-    participant ME as Migration Executor
-    participant DB as PostgreSQL Database
-
-    CLI->>SS: plan("schema.sql")
-    SS->>SP: parseSchemaFile("schema.sql")
-    SP->>SP: Parse CREATE TABLE statements
-    SP-->>SS: Desired Schema (Table[])
-
-    SS->>DB: Connect
-    SS->>DI: getCurrentSchema(client)
-    DI->>DB: Query information_schema
-    DB-->>DI: Table metadata
-    DI-->>SS: Current Schema (Table[])
-
-    SS->>SD: generateMigrationPlan(desired, current)
-    SD->>SD: Compare schemas
-    SD->>SD: Generate SQL statements
-    SD-->>SS: Migration Plan
-
-    alt Command is "plan"
-        SS->>CLI: Display changes
-    else Command is "apply"
-        SS->>ME: executePlan(client, plan)
-        ME->>DB: Execute SQL statements
-        DB-->>ME: Success/Failure
-        ME-->>SS: Results
-    end
-
-    SS->>DB: Disconnect
+```bash
+npm install -g pgterra
 ```
 
-## 📝 Simple Example
-
-Let's walk through a simple example to see how PGTerra works:
+## How it works
 
-### Starting Point: Empty Database
-
-```sql
--- Database has no tables
-```
-
-### Define Desired State: `schema.sql`
+**1. Start with a schema:**
 
 ```sql
+-- schema.sql
 CREATE TABLE users (
-    id SERIAL PRIMARY KEY,
-    email VARCHAR(255) NOT NULL,
-    name VARCHAR(100) NOT NULL,
-    created_at TIMESTAMP DEFAULT NOW()
+  id SERIAL PRIMARY KEY,
+  email VARCHAR(255) NOT NULL UNIQUE,
+  created_at TIMESTAMP DEFAULT NOW()
 );
 ```
 
-### Run Plan Command
-
 ```bash
-pgterra plan
+pgterra apply  # Creates the table
 ```
 
-**Output:**
-
-```
-📋 Analyzing schema changes...
-📝 Found 1 change(s) to apply:
-
-1. CREATE TABLE users (id SERIAL PRIMARY KEY, email VARCHAR(255) NOT NULL, name VARCHAR(100) NOT NULL, created_at TIMESTAMP DEFAULT NOW());
-```
-
-### Apply Changes
-
-```bash
-pgterra apply
-```
-
-**Result:** The `users` table is created in your database.
-
-## 🔄 Schema Evolution Example
-
-Now let's modify the existing table:
-
-### Current Database State:
+**2. Update your schema declaratively:**
 
 ```sql
+-- schema.sql
 CREATE TABLE users (
-    id SERIAL PRIMARY KEY,
-    email VARCHAR(255) NOT NULL,
-    name VARCHAR(100) NOT NULL,
-    created_at TIMESTAMP DEFAULT NOW()
+  id SERIAL PRIMARY KEY,
+  email VARCHAR(255) NOT NULL UNIQUE,
+  full_name VARCHAR(200) NOT NULL,  -- new column
+  is_active BOOLEAN DEFAULT true,   -- another new column
+  created_at TIMESTAMP DEFAULT NOW()
 );
-```
-
-### Update Desired State: `schema.sql`
 
-```sql
-CREATE TABLE users (
-    id SERIAL PRIMARY KEY,
-    email VARCHAR(255) NOT NULL,
-    full_name VARCHAR(200) NOT NULL,    -- renamed and expanded
-    created_at TIMESTAMP DEFAULT NOW(),
-    is_active BOOLEAN DEFAULT true      -- new column
+CREATE TABLE posts (                 -- new table
+  id SERIAL PRIMARY KEY,
+  title VARCHAR(255) NOT NULL,
+  user_id INTEGER REFERENCES users(id),
+  created_at TIMESTAMP DEFAULT NOW()
 );
 ```
 
-### Generated Migration Plan:
-
-```sql
-ALTER TABLE users ADD COLUMN full_name VARCHAR(200) NOT NULL;
-ALTER TABLE users ADD COLUMN is_active BOOLEAN DEFAULT true;
-ALTER TABLE users DROP COLUMN name;
-```
-
-**PGTerra automatically:**
-
-- Detected the new `full_name` column
-- Added the new `is_active` column with default
-- Removed the old `name` column
-- Preserved all existing data
-
-## 🛠️ Advanced Features
-
-### Smart Type Conversions
-
-When changing column types, PGTerra automatically handles complex conversions:
-
-```sql
--- Before: age VARCHAR(10)
--- After:  age INTEGER
-
--- Generated migration:
-ALTER TABLE users ALTER COLUMN age TYPE INTEGER USING age::INTEGER;
-```
-
-### Constraint Management
-
-PGTerra intelligently handles constraint changes:
-
-```mermaid
-flowchart TD
-    A["Column Modification"] --> B["Drop Conflicting Defaults"]
-    B --> C["Change Data Type"]
-    C --> D["Set New Default"]
-    D --> E["Add/Drop NOT NULL"]
-
-    F["Complex Example:"] --> G["VARCHAR → INTEGER<br/>with new DEFAULT"]
-    G --> H["1. DROP DEFAULT"]
-    H --> I["2. ALTER TYPE USING"]
-    I --> J["3. SET DEFAULT"]
-    J --> K["4. SET NOT NULL"]
-```
-
-### Operation Ordering
-
-The differ carefully orders operations to avoid conflicts:
-
-1. Drop conflicting defaults
-2. Change data types (with USING clauses when needed)
-3. Set new defaults
-4. Modify NULL/NOT NULL constraints
-
-## 🚀 Installation & Usage
-
-### Prerequisites
-
-- Node.js 18+ with Bun package manager
-- PostgreSQL database
-- Database connection configured
-
-### Installation
+**3. pgterra calculates the migration:**
 
 ```bash
-bun install
-```
-
-### Commands
-
-#### Plan Changes
+$ pgterra plan
+📋 Analyzing schema changes...
 
-Preview what changes would be made without applying them:
+Planned changes:
+  1. ALTER TABLE users ADD COLUMN full_name VARCHAR(200) NOT NULL
+  2. ALTER TABLE users ADD COLUMN is_active BOOLEAN DEFAULT true  
+  3. CREATE TABLE posts (id SERIAL PRIMARY KEY, title VARCHAR(255) NOT NULL, user_id INTEGER REFERENCES users(id), created_at TIMESTAMP DEFAULT NOW())
 
-```bash
-bun run cli plan
+$ pgterra apply  # Applies the changes safely
 ```
 
-#### Apply Changes
-
-Execute the planned changes:
+**That's it.** No migration files, no manual ALTER statements, no dependency ordering. Just define what you want.
 
-```bash
-bun run cli apply
-```
+## Configuration
 
-#### Use Custom Schema File
+Set your database connection:
 
 ```bash
-bun run cli plan --file custom-schema.sql
-bun run cli apply --file custom-schema.sql
+export DB_HOST=localhost
+export DB_PORT=5432
+export DB_NAME=mydb
+export DB_USER=postgres
+export DB_PASSWORD=password
 ```
 
-### Configuration
-
-Create a configuration file (e.g., `pgterra.config.json`):
-
-```json
-{
-  "database": {
-    "host": "localhost",
-    "port": 5432,
-    "database": "myapp",
-    "username": "postgres",
-    "password": "password"
-  }
-}
-```
-
-## 📁 Project Structure
-
-```
-pgterra/
-├── src/
-│   ├── cli/
-│   │   └── commands/          # CLI command handlers
-│   ├── core/
-│   │   ├── database/          # Database connection & client
-│   │   ├── migration/         # Migration planning & execution
-│   │   └── schema/            # Schema parsing, inspection, diffing
-│   ├── types/                 # TypeScript type definitions
-│   └── utils/                 # Shared utilities
-├── schema.sql                 # Your database schema definition
-└── README.md
-```
-
-## 🔧 Technical Deep Dive
-
-### Schema Differ Logic
+## Features
 
-The `SchemaDiffer` contains the most sophisticated logic:
+- ✅ Tables, columns, and data types
+- ✅ Primary keys, foreign keys, check constraints, unique constraints  
+- ✅ Indexes (btree, gin, gist, partial, expression-based)
+- ✅ ENUM types
+- ✅ Dependency resolution for complex schemas
+- ✅ Data-safe migrations with validation
+- ✅ Destructive operation protection
 
-```mermaid
-flowchart TD
-    A["Compare Schemas"] --> B["Check Tables"]
+## Why declarative?
 
-    B --> C["New Tables"]
-    B --> D["Existing Tables"]
-    B --> E["Dropped Tables"]
+Like Terraform for infrastructure, pgterra lets you define *what* you want, not *how* to get there:
 
-    C --> F["Generate CREATE TABLE"]
-    E --> G["Generate DROP TABLE"]
+- **Version control your complete schema** - not scattered migration files
+- **No migration ordering issues** - pgterra handles dependencies
+- **Easier code reviews** - see the full schema state, not just changes
+- **Safe schema changes** - preview before applying, with rollback support
 
-    D --> H["Compare Columns"]
+## Development
 
-    H --> I["New Columns"]
-    H --> J["Modified Columns"]
-    H --> K["Dropped Columns"]
-
-    I --> L["Generate ADD COLUMN"]
-    K --> M["Generate DROP COLUMN"]
-
-    J --> N["Check What Changed"]
-    N --> O["Data Type"]
-    N --> P["Default Value"]
-    N --> Q["Nullable"]
-
-    O --> R["ALTER COLUMN TYPE<br/>(with USING if needed)"]
-    P --> S["SET/DROP DEFAULT"]
-    Q --> T["SET/DROP NOT NULL"]
-
-    R --> U["Order Operations<br/>Carefully"]
-    S --> U
-    T --> U
-
-    U --> V["Final SQL Statements"]
-```
-
-### Data Structure
-
-Tables and columns are represented as TypeScript interfaces:
-
-```typescript
-interface Column {
-  name: string;
-  type: string;
-  nullable: boolean;
-  default?: string;
-  primary?: boolean;
-}
-
-interface Table {
-  name: string;
-  columns: Column[];
-}
-```
-
-## 🎯 Design Principles
-
-```mermaid
-graph LR
-    A["Declarative<br/>Approach"] --> B["You describe<br/>WHAT you want"]
-    B --> C["Tool figures out<br/>HOW to get there"]
+```bash
+git clone https://github.com/elitan/pgterra.git
+cd pgterra
+bun install
 
-    D["State-Based<br/>Management"] --> E["Compare current vs desired"]
-    E --> F["Generate minimal<br/>change set"]
+# Set up test database connection
+export DATABASE_URL="postgres://user:password@localhost:5432/test_db"
 
-    G["Safety First"] --> H["Preview changes<br/>with 'plan'"]
-    H --> I["Explicit 'apply'<br/>to execute"]
+# Run tests
+bun test
 ```
 
-## 🚦 Current Status & Roadmap
-
-### ✅ **Implemented Features**
-
-#### **Core Table Operations**
-
-- ✅ **Table Creation** - Create new tables from schema definitions
-- ✅ **Table Dropping** - Remove tables not in desired schema
-- ✅ **Mixed Table Operations** - Add, keep, and remove tables in single migration
-
-#### **Column Management**
-
-- ✅ **Column Addition** - Add new columns with various data types
-- ✅ **Column Removal** - Remove columns while preserving data
-- ✅ **Column Type Changes** - Convert between compatible data types
-- ✅ **Default Values** - Handle columns with default values and constraints
-- ✅ **Nullable Constraints** - Manage NOT NULL/NULL constraints
-
-#### **Advanced Type Conversions**
-
-- ✅ **String Type Conversions** - VARCHAR ↔ TEXT with length handling
-- ✅ **Numeric Type Conversions** - INTEGER ↔ BIGINT, DECIMAL precision changes
-- ✅ **Boolean Type Conversions** - All PostgreSQL boolean representations
-- ✅ **Smart USING Clauses** - Automatic type conversion logic
-- ✅ **Unicode Support** - Full Unicode, emoji, and multi-byte character handling
-
-#### **Data Integrity & Safety**
-
-- ✅ **Data Preservation** - All migrations preserve existing data
-- ✅ **Operation Ordering** - Smart ordering to avoid constraint conflicts
-- ✅ **Boundary Value Testing** - Edge cases for all data types
-- ✅ **Large Dataset Support** - Performance-tested with large tables
-
-#### **Performance & Reliability**
-
-- ✅ **Performance Monitoring** - Benchmark tracking and regression detection
-- ✅ **Concurrent Operations** - Lock management and concurrent access
-- ✅ **Memory Efficiency** - Optimized for large datasets
-- ✅ **Error Handling** - Graceful handling of edge cases and failures
-
-#### **CLI & Tooling**
-
-- ✅ **Plan Command** - Preview changes before applying
-- ✅ **Apply Command** - Execute migrations safely
-- ✅ **Schema File Support** - Custom schema file paths
-- ✅ **Database Configuration** - Connection management
-- ✅ **Comprehensive Testing** - 20+ test suites covering edge cases
-
-### 🔄 **In Progress**
-
-#### **Primary Key Support**
-
-- 🔄 **Primary Key Detection** - Currently handles SERIAL PRIMARY KEY
-- 🔄 **Composite Primary Keys** - Multi-column primary keys
-- 🔄 **Primary Key Changes** - Adding/removing/modifying primary keys
-
-### 📋 **Planned Features**
-
-#### **Core Schema Objects**
-
-- [ ] **Indexes**
-  - B-tree, GIN, GiST, BRIN, Hash indexes
-  - Unique, partial, expression indexes
-  - Concurrent creation and REINDEX operations
-- [ ] **Advanced Constraints**
-  - Foreign Keys with CASCADE/RESTRICT/SET NULL actions
-  - Unique Constraints (multi-column)
-  - Check Constraints with custom expressions
-  - DEFERRABLE constraints
-
-#### **Advanced PostgreSQL Features**
-
-- [ ] **Sequences**
-  - Custom sequences with start/increment/min/max
-  - Sequence ownership and dependencies
-- [ ] **Views & Materialized Views**
-  - Standard view creation and OR REPLACE
-  - Materialized view management and refresh
-- [ ] **Custom Types**
-  - ENUMs with value management
-  - Composite Types for complex data structures
-  - Domain types with constraints
-
-#### **Functions & Triggers**
-
-- [ ] **Stored Functions/Procedures**
-  - PL/pgSQL and SQL functions
-  - Parameter and return type management
-  - Function versioning (OR REPLACE)
-- [ ] **Triggers**
-  - BEFORE/AFTER/INSTEAD OF triggers
-  - Row-level and statement-level triggers
-  - Trigger enabling/disabling
-
-#### **Database Administration**
-
-- [ ] **Extensions**
-  - Enable/disable PostgreSQL extensions
-  - Extension version management
-- [ ] **Roles & Security**
-  - User and role management
-  - Permission grants and revokes
-  - Role membership hierarchy
-- [ ] **Schema Namespaces**
-  - Multi-schema support
-  - Schema ownership and search paths
-- [ ] **Tablespaces**
-  - Custom tablespace management
-  - Table and index tablespace assignment
-
-#### **Enhanced Features**
-
-- [ ] **Comments & Documentation**
-  - Object-level comments and descriptions
-  - Schema documentation generation
-- [ ] **Advanced Rules**
-  - Query rewrite rules
-  - Rule creation and management
-- [ ] **Event Triggers**
-  - DDL event triggers
-  - Database-level event handling
-
-#### **Tooling & DevEx Improvements**
-
-- [ ] **Configuration Management**
-  - Multiple environment support
-  - Configuration file formats (JSON/YAML)
-- [ ] **Migration History**
-  - Track applied migrations
-  - Rollback capabilities
-- [ ] **Schema Validation**
-  - Pre-migration validation
-  - Dependency checking
-- [ ] **Import/Export**
-  - Import from existing databases
-  - Export current schema to files
-
-### 📊 **Current Test Coverage**
-
-The project has comprehensive test coverage with **20+ test suites** covering:
-
-- **Table Operations**: 6 core scenarios
-- **Column Operations**: 15+ test suites
-- **Type Conversions**: String, Numeric, Boolean edge cases
-- **Performance Testing**: Large datasets, concurrent operations
-- **Unicode Support**: Emoji, multi-byte characters, escape sequences
-- **Data Integrity**: Boundary values, NULL handling, constraint management
-
-### 🎯 **Next Milestones**
-
-**v0.2.0 - Primary Key & Index Support**
-
-- Complete primary key constraint management
-- Basic index creation and management
-- Performance improvements for large schemas
-
-**v0.3.0 - Foreign Keys & Advanced Constraints**
-
-- Foreign key relationships with actions
-- Unique and check constraints
-- Constraint dependency resolution
-
-**v0.4.0 - Views & Functions**
-
-- Standard and materialized views
-- Basic stored function support
-- Enhanced schema validation
+**Note:** Tests require a `DATABASE_URL` environment variable pointing to a PostgreSQL database. The tests will create and drop tables as needed, so use a dedicated test database.