aios-core 2.1.5 → 2.2.0

package/.aios-core/development/tasks/db-load-csv.md

@@ -1,593 +1,593 @@
-# Task: Load CSV Data Safely
-
-**Purpose**: Import CSV data using PostgreSQL COPY with staging table and validation
-
-**Elicit**: true
-
----
-
-## Execution Modes
-
-**Choose your execution mode:**
-
-### 1. YOLO Mode - Fast, Autonomous (0-1 prompts)
-- Autonomous decision making with logging
-- Minimal user interaction
-- **Best for:** Simple, deterministic tasks
-
-### 2. Interactive Mode - Balanced, Educational (5-10 prompts) **[DEFAULT]**
-- Explicit decision checkpoints
-- Educational explanations
-- **Best for:** Learning, complex decisions
-
-### 3. Pre-Flight Planning - Comprehensive Upfront Planning
-- Task analysis phase (identify all ambiguities)
-- Zero ambiguity execution
-- **Best for:** Ambiguous requirements, critical work
-
-**Parameter:** `mode` (optional, default: `interactive`)
-
----
-
-## Task Definition (AIOS Task Format V1.0)
-
-```yaml
-task: dbLoadCsv()
-responsável: Dara (Sage)
-responsavel_type: Agente
-atomic_layer: Config
-
-**Entrada:**
-- campo: query
-  tipo: string
-  origem: User Input
-  obrigatório: true
-  validação: Valid SQL query
-
-- campo: params
-  tipo: object
-  origem: User Input
-  obrigatório: false
-  validação: Query parameters
-
-- campo: connection
-  tipo: object
-  origem: config
-  obrigatório: true
-  validação: Valid PostgreSQL connection via Supabase
-
-**Saída:**
-- campo: query_result
-  tipo: array
-  destino: Memory
-  persistido: false
-
-- campo: records_affected
-  tipo: number
-  destino: Return value
-  persistido: false
-
-- campo: execution_time
-  tipo: number
-  destino: Memory
-  persistido: false
-```
-
----
-
-## Pre-Conditions
-
-**Purpose:** Validate prerequisites BEFORE task execution (blocking)
-
-**Checklist:**
-
-```yaml
-pre-conditions:
-  - [ ] Database connection established; query syntax valid
-    tipo: pre-condition
-    blocker: true
-    validação: |
-      Check database connection established; query syntax valid
-    error_message: "Pre-condition failed: Database connection established; query syntax valid"
-```
-
----
-
-## Post-Conditions
-
-**Purpose:** Validate execution success AFTER task completes
-
-**Checklist:**
-
-```yaml
-post-conditions:
-  - [ ] Query executed; results returned; transaction committed
-    tipo: post-condition
-    blocker: true
-    validação: |
-      Verify query executed; results returned; transaction committed
-    error_message: "Post-condition failed: Query executed; results returned; transaction committed"
-```
-
----
-
-## Acceptance Criteria
-
-**Purpose:** Definitive pass/fail criteria for task completion
-
-**Checklist:**
-
-```yaml
-acceptance-criteria:
-  - [ ] Data persisted correctly; constraints respected; no orphaned data
-    tipo: acceptance-criterion
-    blocker: true
-    validação: |
-      Assert data persisted correctly; constraints respected; no orphaned data
-    error_message: "Acceptance criterion not met: Data persisted correctly; constraints respected; no orphaned data"
-```
-
----
-
-## Tools
-
-**External/shared resources used by this task:**
-
-- **Tool:** neo4j-driver
-  - **Purpose:** Neo4j database connection and query execution
-  - **Source:** npm: neo4j-driver
-
-- **Tool:** query-validator
-  - **Purpose:** Cypher query syntax validation
-  - **Source:** .aios-core/utils/db-query-validator.js
-
----
-
-## Scripts
-
-**Agent-specific code for this task:**
-
-- **Script:** db-query.js
-  - **Purpose:** Execute Neo4j queries with error handling
-  - **Language:** JavaScript
-  - **Location:** .aios-core/scripts/db-query.js
-
----
-
-## Error Handling
-
-**Strategy:** retry
-
-**Common Errors:**
-
-1. **Error:** Connection Failed
-   - **Cause:** Unable to connect to Neo4j database
-   - **Resolution:** Check connection string, credentials, network
-   - **Recovery:** Retry with exponential backoff (max 3 attempts)
-
-2. **Error:** Query Syntax Error
-   - **Cause:** Invalid Cypher query syntax
-   - **Resolution:** Validate query syntax before execution
-   - **Recovery:** Return detailed syntax error, suggest fix
-
-3. **Error:** Transaction Rollback
-   - **Cause:** Query violates constraints or timeout
-   - **Resolution:** Review query logic and constraints
-   - **Recovery:** Automatic rollback, preserve data integrity
-
----
-
-## Performance
-
-**Expected Metrics:**
-
-```yaml
-duration_expected: 2-10 min (estimated)
-cost_estimated: $0.001-0.008
-token_usage: ~800-2,500 tokens
-```
-
-**Optimization Notes:**
-- Validate configuration early; use atomic writes; implement rollback checkpoints
-
----
-
-## Metadata
-
-```yaml
-story: N/A
-version: 1.0.0
-dependencies:
-  - N/A
-tags:
-  - database
-  - infrastructure
-updated_at: 2025-11-17
-```
-
----
-
-
-## Inputs
-
-- `table` (string): Target table name
-- `csv_file` (string): Path to CSV file
-
----
-
-## Process
-
-### 1. Validate Inputs
-
-Check file exists and table exists:
-
-```bash
-echo "Validating inputs..."
-
-# Check CSV file exists
-[ -f "{csv_file}" ] || {
-  echo "❌ File not found: {csv_file}"
-  exit 1
-}
-
-# Check table exists
-psql "$SUPABASE_DB_URL" -c \
-"SELECT EXISTS (
-  SELECT 1 FROM information_schema.tables
-  WHERE table_schema = 'public' AND table_name = '{table}'
-);" | grep -q t || {
-  echo "❌ Table '{table}' not found"
-  exit 1
-}
-
-# Count CSV rows
-ROW_COUNT=$(wc -l < "{csv_file}" | tr -d ' ')
-echo "✓ CSV file: {csv_file} ($ROW_COUNT rows)"
-echo "✓ Target table: {table}"
-```
-
-### 2. Preview CSV Structure
-
-Show first few rows:
-
-```bash
-echo "CSV Preview (first 5 rows):"
-head -n 5 "{csv_file}"
-echo ""
-echo "Continue with import? (yes/no)"
-read CONFIRM
-[ "$CONFIRM" = "yes" ] || { echo "Aborted"; exit 0; }
-```
-
-### 3. Create Staging Table
-
-Import to staging first for validation:
-
-```bash
-echo "Creating staging table..."
-
-psql "$SUPABASE_DB_URL" << 'EOF'
--- Create staging table with same structure as target
-CREATE TEMP TABLE {table}_staging (LIKE {table} INCLUDING ALL);
-
--- Or if you need to define structure manually:
--- CREATE TEMP TABLE {table}_staging (
---   id TEXT,
---   name TEXT,
---   created_at TEXT
---   -- Define all columns as TEXT initially for flexible parsing
--- );
-
-SELECT 'Staging table created' AS status;
-EOF
-
-echo "✓ Staging table ready"
-```
-
-### 4. COPY Data to Staging
-
-Use PostgreSQL COPY command (fastest method):
-
-```bash
-echo "Loading CSV into staging table..."
-
-# Method 1: Using psql \copy (client-side file)
-psql "$SUPABASE_DB_URL" << 'EOF'
-\copy {table}_staging FROM '{csv_file}' WITH (
-  FORMAT csv,
-  HEADER true,
-  DELIMITER ',',
-  QUOTE '"',
-  ESCAPE '"',
-  NULL 'NULL'
-);
-EOF
-
-# Method 2: Server-side COPY (if file is on server)
-# COPY {table}_staging FROM '/path/to/file.csv' WITH (FORMAT csv, HEADER true);
-
-echo "✓ Data loaded to staging"
-```
-
-### 5. Validate Data
-
-Run validation checks before merging:
-
-```bash
-echo "Validating staged data..."
-
-psql "$SUPABASE_DB_URL" << 'EOF'
--- Check row count
-SELECT COUNT(*) AS staged_rows FROM {table}_staging;
-
--- Check for NULL in required columns (example)
-SELECT COUNT(*) AS null_ids
-FROM {table}_staging
-WHERE id IS NULL;
-
--- Check for duplicates (example)
-SELECT id, COUNT(*) AS duplicates
-FROM {table}_staging
-GROUP BY id
-HAVING COUNT(*) > 1;
-
--- Check data types can be converted (example)
-SELECT
-  COUNT(*) FILTER (WHERE created_at::timestamptz IS NULL) AS invalid_dates
-FROM {table}_staging;
-
--- Any validation failures?
-SELECT
-  CASE
-    WHEN EXISTS (SELECT 1 FROM {table}_staging WHERE id IS NULL) THEN
-      'FAIL: NULL ids found'
-    WHEN EXISTS (SELECT 1 FROM {table}_staging GROUP BY id HAVING COUNT(*) > 1) THEN
-      'FAIL: Duplicate ids found'
-    ELSE
-      'PASS: All validations passed'
-  END AS validation_status;
-EOF
-
-echo ""
-echo "Review validation results above."
-echo "Continue with merge? (yes/no)"
-read CONFIRM
-[ "$CONFIRM" = "yes" ] || { echo "Aborted - data in staging table for review"; exit 1; }
-```
-
-### 6. Merge to Target Table
-
-Use UPSERT pattern for idempotency:
-
-```bash
-echo "Merging to target table..."
-
-psql "$SUPABASE_DB_URL" << 'EOF'
-BEGIN;
-
--- Insert new rows or update existing (idempotent)
-INSERT INTO {table} (id, name, created_at, ...)
-SELECT
-  id::uuid,                  -- Cast to proper types
-  name,
-  created_at::timestamptz,
-  ...
-FROM {table}_staging
-ON CONFLICT (id) DO UPDATE SET
-  name = EXCLUDED.name,
-  created_at = EXCLUDED.created_at,
-  updated_at = NOW();  -- Update timestamp
-
--- Get counts
-SELECT
-  (SELECT COUNT(*) FROM {table}) AS final_count,
-  (SELECT COUNT(*) FROM {table}_staging) AS imported_count;
-
-COMMIT;
-
-SELECT 'Import complete' AS status;
-EOF
-
-echo "✓ Data merged successfully"
-```
-
-### 7. Cleanup
-
-Drop staging table:
-
-```bash
-echo "Cleaning up..."
-
-psql "$SUPABASE_DB_URL" << 'EOF'
-DROP TABLE IF EXISTS {table}_staging;
-EOF
-
-echo "✓ Cleanup complete"
-```
-
----
-
-## Output
-
-Display import summary:
-
-```
-✅ CSV IMPORT COMPLETE
-
-CSV File:       {csv_file}
-Target Table:   {table}
-Rows Imported:  {count}
-Duration:       {duration}s
-
-Validation:
-✓ No NULL in required columns
-✓ No duplicate keys
-✓ All data types valid
-
-Next steps:
-- Verify data in database
-- Run smoke tests if needed
-- Update statistics: ANALYZE {table};
-```
-
----
-
-## Best Practices
-
-### CSV Format Requirements
-
-**Required:**
-- UTF-8 encoding
-- Consistent delimiters (comma recommended)
-- Header row with column names
-- Quoted strings if they contain delimiters
-
-**Example:**
-```csv
-id,name,email,created_at
-"user-1","John Doe","john@example.com","2024-01-01 00:00:00"
-"user-2","Jane Smith","jane@example.com","2024-01-02 00:00:00"
-```
-
-### Handling Large Files
-
-For CSV files > 100MB or > 1M rows:
-
-1. **Split the file:**
-```bash
-split -l 100000 large.csv chunk_
-```
-
-2. **Import in batches:**
-```bash
-for file in chunk_*; do
-  *load-csv {table} $file
-done
-```
-
-3. **Or use streaming COPY:**
-```bash
-cat large.csv | psql "$SUPABASE_DB_URL" -c \
-  "COPY {table} FROM STDIN WITH (FORMAT csv, HEADER true);"
-```
-
-### Data Type Conversion
-
-Always cast from TEXT to proper types in SELECT:
-
-```sql
-SELECT
-  id::uuid,                    -- UUID
-  amount::numeric(10,2),       -- Decimal
-  created_at::timestamptz,     -- Timestamp
-  is_active::boolean,          -- Boolean
-  metadata::jsonb             -- JSON
-FROM {table}_staging
-```
-
----
-
-## Common Issues
-
-### Issue 1: Character Encoding
-
-**Error:** `invalid byte sequence for encoding "UTF8"`
-
-**Fix:**
-```bash
-# Convert to UTF-8
-iconv -f ISO-8859-1 -t UTF-8 input.csv > output.csv
-```
-
-### Issue 2: Quote/Delimiter Conflicts
-
-**Error:** `unterminated CSV quoted field`
-
-**Fix:** Adjust COPY parameters:
-```sql
-COPY table FROM 'file.csv' WITH (
-  DELIMITER ';',    -- Change delimiter
-  QUOTE '''',       -- Change quote character
-  ESCAPE '\'       -- Change escape character
-);
-```
-
-### Issue 3: NULL Values
-
-**Error:** `null value in column "id" violates not-null constraint`
-
-**Fix:** Define NULL representation:
-```sql
-COPY table FROM 'file.csv' WITH (
-  NULL 'NULL',      -- Treat literal "NULL" as NULL
-  -- Or NULL ''     -- Treat empty strings as NULL
-);
-```
-
----
-
-## Security Notes
-
-- **Never** COPY from untrusted sources without validation
-- Always use staging table first
-- Validate data types and constraints before merging
-- Check for SQL injection in CSV content (though COPY is safe)
-- Consider row-level security (RLS) when loading to Supabase
-
----
-
-## Performance Tips
-
-1. **Disable triggers during bulk load:**
-```sql
-ALTER TABLE {table} DISABLE TRIGGER ALL;
--- Load data
-ALTER TABLE {table} ENABLE TRIGGER ALL;
-```
-
-2. **Drop indexes, load, recreate:**
-```sql
--- Only for initial loads, not updates!
-DROP INDEX idx_name;
--- Load data
-CREATE INDEX CONCURRENTLY idx_name ON {table}(column);
-```
-
-3. **Use UNLOGGED tables for staging:**
-```sql
-CREATE UNLOGGED TABLE {table}_staging (...);
--- Faster writes, but not crash-safe
-```
-
-4. **Batch commits:**
-```sql
--- For very large loads
-BEGIN;
-COPY ... -- Load 100k rows
-COMMIT;
-BEGIN;
-COPY ... -- Load next 100k rows
-COMMIT;
-```
-
----
-
-## Alternative: INSERT from Application
-
-For small datasets (<1000 rows), can use regular INSERT:
-
-```javascript
-// Supabase client example
-const { data, error } = await supabase
-  .from('table')
-  .upsert(csvData, { onConflict: 'id' })
-```
-
-But COPY is **10-100x faster** for bulk loads!
-
----
-
-## References
-
-- [PostgreSQL COPY Documentation](https://www.postgresql.org/docs/current/sql-copy.html)
-- [psql \copy Command](https://www.postgresql.org/docs/current/app-psql.html)
+# Task: Load CSV Data Safely
+
+**Purpose**: Import CSV data using PostgreSQL COPY with staging table and validation
+
+**Elicit**: true
+
+---
+
+## Execution Modes
+
+**Choose your execution mode:**
+
+### 1. YOLO Mode - Fast, Autonomous (0-1 prompts)
+- Autonomous decision making with logging
+- Minimal user interaction
+- **Best for:** Simple, deterministic tasks
+
+### 2. Interactive Mode - Balanced, Educational (5-10 prompts) **[DEFAULT]**
+- Explicit decision checkpoints
+- Educational explanations
+- **Best for:** Learning, complex decisions
+
+### 3. Pre-Flight Planning - Comprehensive Upfront Planning
+- Task analysis phase (identify all ambiguities)
+- Zero ambiguity execution
+- **Best for:** Ambiguous requirements, critical work
+
+**Parameter:** `mode` (optional, default: `interactive`)
+
+---
+
+## Task Definition (AIOS Task Format V1.0)
+
+```yaml
+task: dbLoadCsv()
+responsável: Dara (Sage)
+responsavel_type: Agente
+atomic_layer: Config
+
+**Entrada:**
+- campo: query
+  tipo: string
+  origem: User Input
+  obrigatório: true
+  validação: Valid SQL query
+
+- campo: params
+  tipo: object
+  origem: User Input
+  obrigatório: false
+  validação: Query parameters
+
+- campo: connection
+  tipo: object
+  origem: config
+  obrigatório: true
+  validação: Valid PostgreSQL connection via Supabase
+
+**Saída:**
+- campo: query_result
+  tipo: array
+  destino: Memory
+  persistido: false
+
+- campo: records_affected
+  tipo: number
+  destino: Return value
+  persistido: false
+
+- campo: execution_time
+  tipo: number
+  destino: Memory
+  persistido: false
+```
+
+---
+
+## Pre-Conditions
+
+**Purpose:** Validate prerequisites BEFORE task execution (blocking)
+
+**Checklist:**
+
+```yaml
+pre-conditions:
+  - [ ] Database connection established; query syntax valid
+    tipo: pre-condition
+    blocker: true
+    validação: |
+      Check database connection established; query syntax valid
+    error_message: "Pre-condition failed: Database connection established; query syntax valid"
+```
+
+---
+
+## Post-Conditions
+
+**Purpose:** Validate execution success AFTER task completes
+
+**Checklist:**
+
+```yaml
+post-conditions:
+  - [ ] Query executed; results returned; transaction committed
+    tipo: post-condition
+    blocker: true
+    validação: |
+      Verify query executed; results returned; transaction committed
+    error_message: "Post-condition failed: Query executed; results returned; transaction committed"
+```
+
+---
+
+## Acceptance Criteria
+
+**Purpose:** Definitive pass/fail criteria for task completion
+
+**Checklist:**
+
+```yaml
+acceptance-criteria:
+  - [ ] Data persisted correctly; constraints respected; no orphaned data
+    tipo: acceptance-criterion
+    blocker: true
+    validação: |
+      Assert data persisted correctly; constraints respected; no orphaned data
+    error_message: "Acceptance criterion not met: Data persisted correctly; constraints respected; no orphaned data"
+```
+
+---
+
+## Tools
+
+**External/shared resources used by this task:**
+
+- **Tool:** neo4j-driver
+  - **Purpose:** Neo4j database connection and query execution
+  - **Source:** npm: neo4j-driver
+
+- **Tool:** query-validator
+  - **Purpose:** Cypher query syntax validation
+  - **Source:** .aios-core/utils/db-query-validator.js
+
+---
+
+## Scripts
+
+**Agent-specific code for this task:**
+
+- **Script:** db-query.js
+  - **Purpose:** Execute Neo4j queries with error handling
+  - **Language:** JavaScript
+  - **Location:** .aios-core/scripts/db-query.js
+
+---
+
+## Error Handling
+
+**Strategy:** retry
+
+**Common Errors:**
+
+1. **Error:** Connection Failed
+   - **Cause:** Unable to connect to Neo4j database
+   - **Resolution:** Check connection string, credentials, network
+   - **Recovery:** Retry with exponential backoff (max 3 attempts)
+
+2. **Error:** Query Syntax Error
+   - **Cause:** Invalid Cypher query syntax
+   - **Resolution:** Validate query syntax before execution
+   - **Recovery:** Return detailed syntax error, suggest fix
+
+3. **Error:** Transaction Rollback
+   - **Cause:** Query violates constraints or timeout
+   - **Resolution:** Review query logic and constraints
+   - **Recovery:** Automatic rollback, preserve data integrity
+
+---
+
+## Performance
+
+**Expected Metrics:**
+
+```yaml
+duration_expected: 2-10 min (estimated)
+cost_estimated: $0.001-0.008
+token_usage: ~800-2,500 tokens
+```
+
+**Optimization Notes:**
+- Validate configuration early; use atomic writes; implement rollback checkpoints
+
+---
+
+## Metadata
+
+```yaml
+story: N/A
+version: 1.0.0
+dependencies:
+  - N/A
+tags:
+  - database
+  - infrastructure
+updated_at: 2025-11-17
+```
+
+---
+
+
+## Inputs
+
+- `table` (string): Target table name
+- `csv_file` (string): Path to CSV file
+
+---
+
+## Process
+
+### 1. Validate Inputs
+
+Check file exists and table exists:
+
+```bash
+echo "Validating inputs..."
+
+# Check CSV file exists
+[ -f "{csv_file}" ] || {
+  echo "❌ File not found: {csv_file}"
+  exit 1
+}
+
+# Check table exists
+psql "$SUPABASE_DB_URL" -c \
+"SELECT EXISTS (
+  SELECT 1 FROM information_schema.tables
+  WHERE table_schema = 'public' AND table_name = '{table}'
+);" | grep -q t || {
+  echo "❌ Table '{table}' not found"
+  exit 1
+}
+
+# Count CSV rows
+ROW_COUNT=$(wc -l < "{csv_file}" | tr -d ' ')
+echo "✓ CSV file: {csv_file} ($ROW_COUNT rows)"
+echo "✓ Target table: {table}"
+```
+
+### 2. Preview CSV Structure
+
+Show first few rows:
+
+```bash
+echo "CSV Preview (first 5 rows):"
+head -n 5 "{csv_file}"
+echo ""
+echo "Continue with import? (yes/no)"
+read CONFIRM
+[ "$CONFIRM" = "yes" ] || { echo "Aborted"; exit 0; }
+```
+
+### 3. Create Staging Table
+
+Import to staging first for validation:
+
+```bash
+echo "Creating staging table..."
+
+psql "$SUPABASE_DB_URL" << 'EOF'
+-- Create staging table with same structure as target
+CREATE TEMP TABLE {table}_staging (LIKE {table} INCLUDING ALL);
+
+-- Or if you need to define structure manually:
+-- CREATE TEMP TABLE {table}_staging (
+--   id TEXT,
+--   name TEXT,
+--   created_at TEXT
+--   -- Define all columns as TEXT initially for flexible parsing
+-- );
+
+SELECT 'Staging table created' AS status;
+EOF
+
+echo "✓ Staging table ready"
+```
+
+### 4. COPY Data to Staging
+
+Use PostgreSQL COPY command (fastest method):
+
+```bash
+echo "Loading CSV into staging table..."
+
+# Method 1: Using psql \copy (client-side file)
+psql "$SUPABASE_DB_URL" << 'EOF'
+\copy {table}_staging FROM '{csv_file}' WITH (
+  FORMAT csv,
+  HEADER true,
+  DELIMITER ',',
+  QUOTE '"',
+  ESCAPE '"',
+  NULL 'NULL'
+);
+EOF
+
+# Method 2: Server-side COPY (if file is on server)
+# COPY {table}_staging FROM '/path/to/file.csv' WITH (FORMAT csv, HEADER true);
+
+echo "✓ Data loaded to staging"
+```
+
+### 5. Validate Data
+
+Run validation checks before merging:
+
+```bash
+echo "Validating staged data..."
+
+psql "$SUPABASE_DB_URL" << 'EOF'
+-- Check row count
+SELECT COUNT(*) AS staged_rows FROM {table}_staging;
+
+-- Check for NULL in required columns (example)
+SELECT COUNT(*) AS null_ids
+FROM {table}_staging
+WHERE id IS NULL;
+
+-- Check for duplicates (example)
+SELECT id, COUNT(*) AS duplicates
+FROM {table}_staging
+GROUP BY id
+HAVING COUNT(*) > 1;
+
+-- Check data types can be converted (example)
+SELECT
+  COUNT(*) FILTER (WHERE created_at::timestamptz IS NULL) AS invalid_dates
+FROM {table}_staging;
+
+-- Any validation failures?
+SELECT
+  CASE
+    WHEN EXISTS (SELECT 1 FROM {table}_staging WHERE id IS NULL) THEN
+      'FAIL: NULL ids found'
+    WHEN EXISTS (SELECT 1 FROM {table}_staging GROUP BY id HAVING COUNT(*) > 1) THEN
+      'FAIL: Duplicate ids found'
+    ELSE
+      'PASS: All validations passed'
+  END AS validation_status;
+EOF
+
+echo ""
+echo "Review validation results above."
+echo "Continue with merge? (yes/no)"
+read CONFIRM
+[ "$CONFIRM" = "yes" ] || { echo "Aborted - data in staging table for review"; exit 1; }
+```
+
+### 6. Merge to Target Table
+
+Use UPSERT pattern for idempotency:
+
+```bash
+echo "Merging to target table..."
+
+psql "$SUPABASE_DB_URL" << 'EOF'
+BEGIN;
+
+-- Insert new rows or update existing (idempotent)
+INSERT INTO {table} (id, name, created_at, ...)
+SELECT
+  id::uuid,                  -- Cast to proper types
+  name,
+  created_at::timestamptz,
+  ...
+FROM {table}_staging
+ON CONFLICT (id) DO UPDATE SET
+  name = EXCLUDED.name,
+  created_at = EXCLUDED.created_at,
+  updated_at = NOW();  -- Update timestamp
+
+-- Get counts
+SELECT
+  (SELECT COUNT(*) FROM {table}) AS final_count,
+  (SELECT COUNT(*) FROM {table}_staging) AS imported_count;
+
+COMMIT;
+
+SELECT 'Import complete' AS status;
+EOF
+
+echo "✓ Data merged successfully"
+```
+
+### 7. Cleanup
+
+Drop staging table:
+
+```bash
+echo "Cleaning up..."
+
+psql "$SUPABASE_DB_URL" << 'EOF'
+DROP TABLE IF EXISTS {table}_staging;
+EOF
+
+echo "✓ Cleanup complete"
+```
+
+---
+
+## Output
+
+Display import summary:
+
+```
+✅ CSV IMPORT COMPLETE
+
+CSV File:       {csv_file}
+Target Table:   {table}
+Rows Imported:  {count}
+Duration:       {duration}s
+
+Validation:
+✓ No NULL in required columns
+✓ No duplicate keys
+✓ All data types valid
+
+Next steps:
+- Verify data in database
+- Run smoke tests if needed
+- Update statistics: ANALYZE {table};
+```
+
+---
+
+## Best Practices
+
+### CSV Format Requirements
+
+**Required:**
+- UTF-8 encoding
+- Consistent delimiters (comma recommended)
+- Header row with column names
+- Quoted strings if they contain delimiters
+
+**Example:**
+```csv
+id,name,email,created_at
+"user-1","John Doe","john@example.com","2024-01-01 00:00:00"
+"user-2","Jane Smith","jane@example.com","2024-01-02 00:00:00"
+```
+
+### Handling Large Files
+
+For CSV files > 100MB or > 1M rows:
+
+1. **Split the file:**
+```bash
+split -l 100000 large.csv chunk_
+```
+
+2. **Import in batches:**
+```bash
+for file in chunk_*; do
+  *load-csv {table} $file
+done
+```
+
+3. **Or use streaming COPY:**
+```bash
+cat large.csv | psql "$SUPABASE_DB_URL" -c \
+  "COPY {table} FROM STDIN WITH (FORMAT csv, HEADER true);"
+```
+
+### Data Type Conversion
+
+Always cast from TEXT to proper types in SELECT:
+
+```sql
+SELECT
+  id::uuid,                    -- UUID
+  amount::numeric(10,2),       -- Decimal
+  created_at::timestamptz,     -- Timestamp
+  is_active::boolean,          -- Boolean
+  metadata::jsonb             -- JSON
+FROM {table}_staging
+```
+
+---
+
+## Common Issues
+
+### Issue 1: Character Encoding
+
+**Error:** `invalid byte sequence for encoding "UTF8"`
+
+**Fix:**
+```bash
+# Convert to UTF-8
+iconv -f ISO-8859-1 -t UTF-8 input.csv > output.csv
+```
+
+### Issue 2: Quote/Delimiter Conflicts
+
+**Error:** `unterminated CSV quoted field`
+
+**Fix:** Adjust COPY parameters:
+```sql
+COPY table FROM 'file.csv' WITH (
+  DELIMITER ';',    -- Change delimiter
+  QUOTE '''',       -- Change quote character
+  ESCAPE '\'       -- Change escape character
+);
+```
+
+### Issue 3: NULL Values
+
+**Error:** `null value in column "id" violates not-null constraint`
+
+**Fix:** Define NULL representation:
+```sql
+COPY table FROM 'file.csv' WITH (
+  NULL 'NULL',      -- Treat literal "NULL" as NULL
+  -- Or NULL ''     -- Treat empty strings as NULL
+);
+```
+
+---
+
+## Security Notes
+
+- **Never** COPY from untrusted sources without validation
+- Always use staging table first
+- Validate data types and constraints before merging
+- Check for SQL injection in CSV content (though COPY is safe)
+- Consider row-level security (RLS) when loading to Supabase
+
+---
+
+## Performance Tips
+
+1. **Disable triggers during bulk load:**
+```sql
+ALTER TABLE {table} DISABLE TRIGGER ALL;
+-- Load data
+ALTER TABLE {table} ENABLE TRIGGER ALL;
+```
+
+2. **Drop indexes, load, recreate:**
+```sql
+-- Only for initial loads, not updates!
+DROP INDEX idx_name;
+-- Load data
+CREATE INDEX CONCURRENTLY idx_name ON {table}(column);
+```
+
+3. **Use UNLOGGED tables for staging:**
+```sql
+CREATE UNLOGGED TABLE {table}_staging (...);
+-- Faster writes, but not crash-safe
+```
+
+4. **Batch commits:**
+```sql
+-- For very large loads
+BEGIN;
+COPY ... -- Load 100k rows
+COMMIT;
+BEGIN;
+COPY ... -- Load next 100k rows
+COMMIT;
+```
+
+---
+
+## Alternative: INSERT from Application
+
+For small datasets (<1000 rows), can use regular INSERT:
+
+```javascript
+// Supabase client example
+const { data, error } = await supabase
+  .from('table')
+  .upsert(csvData, { onConflict: 'id' })
+```
+
+But COPY is **10-100x faster** for bulk loads!
+
+---
+
+## References
+
+- [PostgreSQL COPY Documentation](https://www.postgresql.org/docs/current/sql-copy.html)
+- [psql \copy Command](https://www.postgresql.org/docs/current/app-psql.html)