aios-core 2.1.0

package/.aios-core/development/tasks/db-load-csv.md

@@ -0,0 +1,593 @@
+# Task: Load CSV Data Safely
+
+**Purpose**: Import CSV data using PostgreSQL COPY with staging table and validation
+
+**Elicit**: true
+
+---
+
+## Execution Modes
+
+**Choose your execution mode:**
+
+### 1. YOLO Mode - Fast, Autonomous (0-1 prompts)
+- Autonomous decision making with logging
+- Minimal user interaction
+- **Best for:** Simple, deterministic tasks
+
+### 2. Interactive Mode - Balanced, Educational (5-10 prompts) **[DEFAULT]**
+- Explicit decision checkpoints
+- Educational explanations
+- **Best for:** Learning, complex decisions
+
+### 3. Pre-Flight Planning - Comprehensive Upfront Planning
+- Task analysis phase (identify all ambiguities)
+- Zero ambiguity execution
+- **Best for:** Ambiguous requirements, critical work
+
+**Parameter:** `mode` (optional, default: `interactive`)
+
+---
+
+## Task Definition (AIOS Task Format V1.0)
+
+```yaml
+task: dbLoadCsv()
+responsável: Dara (Sage)
+responsavel_type: Agente
+atomic_layer: Config
+
+**Entrada:**
+- campo: query
+  tipo: string
+  origem: User Input
+  obrigatório: true
+  validação: Valid SQL query
+
+- campo: params
+  tipo: object
+  origem: User Input
+  obrigatório: false
+  validação: Query parameters
+
+- campo: connection
+  tipo: object
+  origem: config
+  obrigatório: true
+  validação: Valid PostgreSQL connection via Supabase
+
+**Saída:**
+- campo: query_result
+  tipo: array
+  destino: Memory
+  persistido: false
+
+- campo: records_affected
+  tipo: number
+  destino: Return value
+  persistido: false
+
+- campo: execution_time
+  tipo: number
+  destino: Memory
+  persistido: false
+```
+
+---
+
+## Pre-Conditions
+
+**Purpose:** Validate prerequisites BEFORE task execution (blocking)
+
+**Checklist:**
+
+```yaml
+pre-conditions:
+  - [ ] Database connection established; query syntax valid
+    tipo: pre-condition
+    blocker: true
+    validação: |
+      Check database connection established; query syntax valid
+    error_message: "Pre-condition failed: Database connection established; query syntax valid"
+```
+
+---
+
+## Post-Conditions
+
+**Purpose:** Validate execution success AFTER task completes
+
+**Checklist:**
+
+```yaml
+post-conditions:
+  - [ ] Query executed; results returned; transaction committed
+    tipo: post-condition
+    blocker: true
+    validação: |
+      Verify query executed; results returned; transaction committed
+    error_message: "Post-condition failed: Query executed; results returned; transaction committed"
+```
+
+---
+
+## Acceptance Criteria
+
+**Purpose:** Definitive pass/fail criteria for task completion
+
+**Checklist:**
+
+```yaml
+acceptance-criteria:
+  - [ ] Data persisted correctly; constraints respected; no orphaned data
+    tipo: acceptance-criterion
+    blocker: true
+    validação: |
+      Assert data persisted correctly; constraints respected; no orphaned data
+    error_message: "Acceptance criterion not met: Data persisted correctly; constraints respected; no orphaned data"
+```
+
+---
+
+## Tools
+
+**External/shared resources used by this task:**
+
+- **Tool:** neo4j-driver
+  - **Purpose:** Neo4j database connection and query execution
+  - **Source:** npm: neo4j-driver
+
+- **Tool:** query-validator
+  - **Purpose:** Cypher query syntax validation
+  - **Source:** .aios-core/utils/db-query-validator.js
+
+---
+
+## Scripts
+
+**Agent-specific code for this task:**
+
+- **Script:** db-query.js
+  - **Purpose:** Execute Neo4j queries with error handling
+  - **Language:** JavaScript
+  - **Location:** .aios-core/scripts/db-query.js
+
+---
+
+## Error Handling
+
+**Strategy:** retry
+
+**Common Errors:**
+
+1. **Error:** Connection Failed
+   - **Cause:** Unable to connect to Neo4j database
+   - **Resolution:** Check connection string, credentials, network
+   - **Recovery:** Retry with exponential backoff (max 3 attempts)
+
+2. **Error:** Query Syntax Error
+   - **Cause:** Invalid Cypher query syntax
+   - **Resolution:** Validate query syntax before execution
+   - **Recovery:** Return detailed syntax error, suggest fix
+
+3. **Error:** Transaction Rollback
+   - **Cause:** Query violates constraints or timeout
+   - **Resolution:** Review query logic and constraints
+   - **Recovery:** Automatic rollback, preserve data integrity
+
+---
+
+## Performance
+
+**Expected Metrics:**
+
+```yaml
+duration_expected: 2-10 min (estimated)
+cost_estimated: $0.001-0.008
+token_usage: ~800-2,500 tokens
+```
+
+**Optimization Notes:**
+- Validate configuration early; use atomic writes; implement rollback checkpoints
+
+---
+
+## Metadata
+
+```yaml
+story: N/A
+version: 1.0.0
+dependencies:
+  - N/A
+tags:
+  - database
+  - infrastructure
+updated_at: 2025-11-17
+```
+
+---
+
+
+## Inputs
+
+- `table` (string): Target table name
+- `csv_file` (string): Path to CSV file
+
+---
+
+## Process
+
+### 1. Validate Inputs
+
+Check file exists and table exists:
+
+```bash
+echo "Validating inputs..."
+
+# Check CSV file exists
+[ -f "{csv_file}" ] || {
+  echo "❌ File not found: {csv_file}"
+  exit 1
+}
+
+# Check table exists
+psql "$SUPABASE_DB_URL" -c \
+"SELECT EXISTS (
+  SELECT 1 FROM information_schema.tables
+  WHERE table_schema = 'public' AND table_name = '{table}'
+);" | grep -q t || {
+  echo "❌ Table '{table}' not found"
+  exit 1
+}
+
+# Count CSV rows
+ROW_COUNT=$(wc -l < "{csv_file}" | tr -d ' ')
+echo "✓ CSV file: {csv_file} ($ROW_COUNT rows)"
+echo "✓ Target table: {table}"
+```
+
+### 2. Preview CSV Structure
+
+Show first few rows:
+
+```bash
+echo "CSV Preview (first 5 rows):"
+head -n 5 "{csv_file}"
+echo ""
+echo "Continue with import? (yes/no)"
+read CONFIRM
+[ "$CONFIRM" = "yes" ] || { echo "Aborted"; exit 0; }
+```
+
+### 3. Create Staging Table
+
+Import to staging first for validation:
+
+```bash
+echo "Creating staging table..."
+
+psql "$SUPABASE_DB_URL" << 'EOF'
+-- Create staging table with same structure as target
+CREATE TEMP TABLE {table}_staging (LIKE {table} INCLUDING ALL);
+
+-- Or if you need to define structure manually:
+-- CREATE TEMP TABLE {table}_staging (
+--   id TEXT,
+--   name TEXT,
+--   created_at TEXT
+--   -- Define all columns as TEXT initially for flexible parsing
+-- );
+
+SELECT 'Staging table created' AS status;
+EOF
+
+echo "✓ Staging table ready"
+```
+
+### 4. COPY Data to Staging
+
+Use PostgreSQL COPY command (fastest method):
+
+```bash
+echo "Loading CSV into staging table..."
+
+# Method 1: Using psql \copy (client-side file)
+psql "$SUPABASE_DB_URL" << 'EOF'
+\copy {table}_staging FROM '{csv_file}' WITH (
+  FORMAT csv,
+  HEADER true,
+  DELIMITER ',',
+  QUOTE '"',
+  ESCAPE '"',
+  NULL 'NULL'
+);
+EOF
+
+# Method 2: Server-side COPY (if file is on server)
+# COPY {table}_staging FROM '/path/to/file.csv' WITH (FORMAT csv, HEADER true);
+
+echo "✓ Data loaded to staging"
+```
+
+### 5. Validate Data
+
+Run validation checks before merging:
+
+```bash
+echo "Validating staged data..."
+
+psql "$SUPABASE_DB_URL" << 'EOF'
+-- Check row count
+SELECT COUNT(*) AS staged_rows FROM {table}_staging;
+
+-- Check for NULL in required columns (example)
+SELECT COUNT(*) AS null_ids
+FROM {table}_staging
+WHERE id IS NULL;
+
+-- Check for duplicates (example)
+SELECT id, COUNT(*) AS duplicates
+FROM {table}_staging
+GROUP BY id
+HAVING COUNT(*) > 1;
+
+-- Check data types can be converted (example)
+SELECT
+  COUNT(*) FILTER (WHERE created_at::timestamptz IS NULL) AS invalid_dates
+FROM {table}_staging;
+
+-- Any validation failures?
+SELECT
+  CASE
+    WHEN EXISTS (SELECT 1 FROM {table}_staging WHERE id IS NULL) THEN
+      'FAIL: NULL ids found'
+    WHEN EXISTS (SELECT 1 FROM {table}_staging GROUP BY id HAVING COUNT(*) > 1) THEN
+      'FAIL: Duplicate ids found'
+    ELSE
+      'PASS: All validations passed'
+  END AS validation_status;
+EOF
+
+echo ""
+echo "Review validation results above."
+echo "Continue with merge? (yes/no)"
+read CONFIRM
+[ "$CONFIRM" = "yes" ] || { echo "Aborted - data in staging table for review"; exit 1; }
+```
+
+### 6. Merge to Target Table
+
+Use UPSERT pattern for idempotency:
+
+```bash
+echo "Merging to target table..."
+
+psql "$SUPABASE_DB_URL" << 'EOF'
+BEGIN;
+
+-- Insert new rows or update existing (idempotent)
+INSERT INTO {table} (id, name, created_at, ...)
+SELECT
+  id::uuid,                  -- Cast to proper types
+  name,
+  created_at::timestamptz,
+  ...
+FROM {table}_staging
+ON CONFLICT (id) DO UPDATE SET
+  name = EXCLUDED.name,
+  created_at = EXCLUDED.created_at,
+  updated_at = NOW();  -- Update timestamp
+
+-- Get counts
+SELECT
+  (SELECT COUNT(*) FROM {table}) AS final_count,
+  (SELECT COUNT(*) FROM {table}_staging) AS imported_count;
+
+COMMIT;
+
+SELECT 'Import complete' AS status;
+EOF
+
+echo "✓ Data merged successfully"
+```
+
+### 7. Cleanup
+
+Drop staging table:
+
+```bash
+echo "Cleaning up..."
+
+psql "$SUPABASE_DB_URL" << 'EOF'
+DROP TABLE IF EXISTS {table}_staging;
+EOF
+
+echo "✓ Cleanup complete"
+```
+
+---
+
+## Output
+
+Display import summary:
+
+```
+✅ CSV IMPORT COMPLETE
+
+CSV File:       {csv_file}
+Target Table:   {table}
+Rows Imported:  {count}
+Duration:       {duration}s
+
+Validation:
+✓ No NULL in required columns
+✓ No duplicate keys
+✓ All data types valid
+
+Next steps:
+- Verify data in database
+- Run smoke tests if needed
+- Update statistics: ANALYZE {table};
+```
+
+---
+
+## Best Practices
+
+### CSV Format Requirements
+
+**Required:**
+- UTF-8 encoding
+- Consistent delimiters (comma recommended)
+- Header row with column names
+- Quoted strings if they contain delimiters
+
+**Example:**
+```csv
+id,name,email,created_at
+"user-1","John Doe","john@example.com","2024-01-01 00:00:00"
+"user-2","Jane Smith","jane@example.com","2024-01-02 00:00:00"
+```
+
+### Handling Large Files
+
+For CSV files > 100MB or > 1M rows:
+
+1. **Split the file:**
+```bash
+split -l 100000 large.csv chunk_
+```
+
+2. **Import in batches:**
+```bash
+for file in chunk_*; do
+  *load-csv {table} $file
+done
+```
+
+3. **Or use streaming COPY:**
+```bash
+cat large.csv | psql "$SUPABASE_DB_URL" -c \
+  "COPY {table} FROM STDIN WITH (FORMAT csv, HEADER true);"
+```
+
+### Data Type Conversion
+
+Always cast from TEXT to proper types in SELECT:
+
+```sql
+SELECT
+  id::uuid,                    -- UUID
+  amount::numeric(10,2),       -- Decimal
+  created_at::timestamptz,     -- Timestamp
+  is_active::boolean,          -- Boolean
+  metadata::jsonb             -- JSON
+FROM {table}_staging
+```
+
+---
+
+## Common Issues
+
+### Issue 1: Character Encoding
+
+**Error:** `invalid byte sequence for encoding "UTF8"`
+
+**Fix:**
+```bash
+# Convert to UTF-8
+iconv -f ISO-8859-1 -t UTF-8 input.csv > output.csv
+```
+
+### Issue 2: Quote/Delimiter Conflicts
+
+**Error:** `unterminated CSV quoted field`
+
+**Fix:** Adjust COPY parameters:
+```sql
+COPY table FROM 'file.csv' WITH (
+  DELIMITER ';',    -- Change delimiter
+  QUOTE '''',       -- Change quote character
+  ESCAPE '\'       -- Change escape character
+);
+```
+
+### Issue 3: NULL Values
+
+**Error:** `null value in column "id" violates not-null constraint`
+
+**Fix:** Define NULL representation:
+```sql
+COPY table FROM 'file.csv' WITH (
+  NULL 'NULL',      -- Treat literal "NULL" as NULL
+  -- Or NULL ''     -- Treat empty strings as NULL
+);
+```
+
+---
+
+## Security Notes
+
+- **Never** COPY from untrusted sources without validation
+- Always use staging table first
+- Validate data types and constraints before merging
+- Check for SQL injection in CSV content (though COPY is safe)
+- Consider row-level security (RLS) when loading to Supabase
+
+---
+
+## Performance Tips
+
+1. **Disable triggers during bulk load:**
+```sql
+ALTER TABLE {table} DISABLE TRIGGER ALL;
+-- Load data
+ALTER TABLE {table} ENABLE TRIGGER ALL;
+```
+
+2. **Drop indexes, load, recreate:**
+```sql
+-- Only for initial loads, not updates!
+DROP INDEX idx_name;
+-- Load data
+CREATE INDEX CONCURRENTLY idx_name ON {table}(column);
+```
+
+3. **Use UNLOGGED tables for staging:**
+```sql
+CREATE UNLOGGED TABLE {table}_staging (...);
+-- Faster writes, but not crash-safe
+```
+
+4. **Batch commits:**
+```sql
+-- For very large loads
+BEGIN;
+COPY ... -- Load 100k rows
+COMMIT;
+BEGIN;
+COPY ... -- Load next 100k rows
+COMMIT;
+```
+
+---
+
+## Alternative: INSERT from Application
+
+For small datasets (<1000 rows), can use regular INSERT:
+
+```javascript
+// Supabase client example
+const { data, error } = await supabase
+  .from('table')
+  .upsert(csvData, { onConflict: 'id' })
+```
+
+But COPY is **10-100x faster** for bulk loads!
+
+---
+
+## References
+
+- [PostgreSQL COPY Documentation](https://www.postgresql.org/docs/current/sql-copy.html)
+- [psql \copy Command](https://www.postgresql.org/docs/current/app-psql.html)