@trieungoctam/vibekit 1.0.0

package/skills/dev/databases/db-design.md

@@ -0,0 +1,188 @@
+---
+name: db-design
+description: "Activate when user requests: Database/table design for transactional (OLTP), analytics (OLAP), create or extend schema, design fact/dimension tables, analyze/review CSV/JSON/SQL files to create tables, or need advice on data storage structure."
+---
+
+# SKILL: Database Schema Design
+
+## Execution Process
+
+### Step 1: Analyze Context (MANDATORY)
+
+Before ANY design work, AI MUST check:
+
+- [ ] Database stack identified? (MySQL / PostgreSQL / BigQuery / SQLite / D1)
+- [ ] Existing schema explored via tools/MCP?
+- [ ] Use case clear? (transactional vs analytics vs ETL)
+
+**If ANY checkbox is unclear → STOP and ask user before proceeding.**
+
+### Step 2: Gather Requirements
+
+**MUST ask if missing:**
+
+| Missing Info | Required Question |
+|--------------|-------------------|
+| Entities unclear | "What are the main entities involved in this feature?" |
+| Query patterns unknown | "What queries/reports will run most often?" |
+| Granularity unclear | "What level of detail needed? (per order, per item, per day)" |
+| Constraints unknown | "Any uniqueness, retention, or multi-tenant requirements?" |
+| Relationships unclear | "How do these entities relate to each other?" |
+
+**DO NOT assume or proceed without answers. DO NOT design schema without understanding the business context.**
+
+### Step 3: Load Documentation by Intent
+
+| Intent | Keywords | Load file |
+|--------|----------|-----------|
+| **OLTP** | "business tables", "transactional", "CRUD", "orders", "users" | [transactional.md](transactional.md) |
+| **OLAP** | "analytics", "statistics", "reports", "fact", "dimension" | [analytics.md](analytics.md) |
+| **ETL** | "incremental", "load", "sync", "watermark", "etl" | [incremental-etl.md](incremental-etl.md) |
+
+**Database stacks:** [MySQL](stacks/mysql.md) · [PostgreSQL](stacks/postgres.md) · [BigQuery](stacks/bigquery.md) · [SQLite](stacks/sqlite.md) · [D1 Cloudflare](stacks/d1_cloudflare.md)
+
+### Step 4: Design & Propose
+
+Only after Steps 1-3 complete:
+- Propose DDL with reasoning
+- **Wait for user approval before execution**
+- Never auto-execute DDL
+
+### Step 5: Verify Checklist
+
+Before delivering final DDL, verify the **Final Checklist** at the end of this file.
+
+---
+
+## Scope & Objectives
+
+Design or extend database schemas ensuring:
+
+- Consistent, understandable, maintainable data structures
+- Good performance with appropriate indexes
+- Design that fits existing tables (avoid duplication, avoid breaking logic)
+
+---
+
+## Naming Conventions
+
+### Table naming
+
+- Use **snake_case**, English, plural for main entities:
+  - `users`, `orders`, `order_items`, `product_variants`
+- Junction/mapping tables:
+  - `role_permissions`, `user_roles`, `product_categories`
+- Fact & dimension tables (analytics):
+  - Fact: `fact_orders`, `fact_order_items`, `fact_pageviews`
+  - Dimension: `dim_date`, `dim_customer`, `dim_product`, `dim_store`
+- Avoid unclear abbreviations:
+  - ❌ `usr`, `ord`, `inv_tx`
+  - ✅ `users`, `orders`, `inventory_transactions`
+
+### Column naming
+
+- Use **snake_case**, clear meaning, consistent across tables
+- **Primary key**: `id` (main PK of table)
+- **Foreign key**: `xxx_id` pointing to table `xxx`:
+  - `user_id` → `users.id`
+  - `order_id` → `orders.id`
+- **Audit & lifecycle**:
+  - `created_at`, `updated_at`, `deleted_at` (soft delete)
+- **Boolean flags**:
+  - `is_active`, `is_deleted`, `is_primary`, `is_verified`
+- **Quantities & prices**:
+  - `quantity`, `unit_price`, `total_price`, `discount_amount`, `tax_amount`
+- **Time ranges**:
+  - `start_at`, `end_at` or `valid_from`, `valid_to` (pick one convention and use consistently)
+- **Enum/state**:
+  - `status` (combine with ENUM / CHECK constraint)
+
+---
+
+## Workflow
+
+### 1. Understand business requirements
+
+- Summarize requirements in your own words
+- If missing context, ask:
+  - What entities are involved?
+  - What is the main business flow?
+  - What queries/reports are commonly used?
+
+### 2. Explore existing schema (MANDATORY before proposing)
+
+- Use tools/MCP to list tables and describe structure
+- For DBs with comments (MySQL/PostgreSQL/BigQuery): read comments directly
+- For SQLite/D1: query `metadata_tables` and `metadata_columns` tables
+
+### 3. Analyze and propose draft
+
+- Which tables already partially cover requirements?
+- Which tables need additional columns/indexes?
+- Which tables are completely missing (need to create)?
+
+### 4. Confirm with user before finalizing design
+
+- Required granularity
+- Metrics to calculate
+- Important business constraints (uniqueness, retention, privacy)
+
+### 5. Design detailed schema
+
+- Provide proposed DDL - **do NOT auto-execute**
+- Wait for user approval before execution
+
+---
+
+## Processing Data Files (CSV/JSON/SQL)
+
+When user provides files:
+
+1. **Read and identify structure**: headers, data types, relationships
+2. **Large files**: write Python script to parse and analyze
+3. **SQL files**: review naming, indexes, constraints, propose improvements
+4. **Output**: Proposed DDL with reasoning
+
+---
+
+## Output Format
+
+When delivering design results:
+
+1. **Summary**: list of new tables / tables to modify
+2. **Complete DDL SQL**: columns, types, PK, FK, indexes, comments
+3. **Reasoning**: explain why this structure/index was chosen, tied to actual queries
+
+---
+
+## Final Checklist
+
+Before delivering final DDL:
+
+- [ ] Explored existing schema via tools/MCP
+- [ ] No duplicate tables with existing schema
+- [ ] All FKs have indexes
+- [ ] Composite indexes match main query patterns
+- [ ] Naming convention consistent with existing schema
+- [ ] Added comments/descriptions for tables and columns
+- [ ] For SQLite/D1: prepared INSERTs for metadata tables
+- [ ] Explained reasoning for important design decisions
+
+---
+
+## File Structure
+
+```
+.claude/skills/databases/
+├── SKILL.md                  # Skill description (entry + execution process + checklist)
+├── db-design.md              # This file 
+├── transactional.md          # OLTP design rules
+├── analytics.md              # OLAP / Star Schema rules
+├── incremental-etl.md        # ETL & watermark patterns
+└── stacks/
+    ├── mysql.md              # MySQL-specific rules
+    ├── postgres.md           # PostgreSQL-specific rules
+    ├── bigquery.md           # BigQuery-specific rules
+    ├── sqlite.md             # SQLite-specific rules
+    └── d1_cloudflare.md      # D1 Cloudflare-specific rules
+```

package/skills/dev/databases/incremental-etl.md

@@ -0,0 +1,213 @@
+# Incremental Loading & ETL Logs
+
+> **Note:** Core naming conventions, workflow, and checklist are in `SKILL.md` or `db-design.md` (always loaded).
+
+Guidelines for designing schemas to support periodic data loading with tracking and control.
+
+---
+
+## 1. Source Tables Requirements
+
+All source tables that need to support incremental loading must have:
+
+```sql
+-- Required audit columns
+created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
+updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
+
+-- Index for incremental query
+CREATE INDEX idx_tablename_updated_at ON tablename(updated_at);
+```
+
+### Soft delete support
+
+If source has soft delete, need to track deleted records too:
+
+```sql
+deleted_at TIMESTAMP NULL
+
+-- Index to get all modified records (update or delete)
+CREATE INDEX idx_tablename_modified ON tablename(updated_at);
+```
+
+---
+
+## 2. ETL Control Tables
+
+### etl_jobs - Job definitions
+
+```sql
+CREATE TABLE etl_jobs (
+    job_id VARCHAR(100) PRIMARY KEY,
+    job_name VARCHAR(255) NOT NULL,
+    description TEXT,
+    source_table VARCHAR(255),
+    target_table VARCHAR(255),
+    load_type VARCHAR(50) NOT NULL,         -- 'full', 'incremental', 'cdc'
+    schedule VARCHAR(100),                   -- Cron: '0 */1 * * *'
+    is_active BOOLEAN DEFAULT TRUE,
+    timeout_seconds INT DEFAULT 3600,
+    retry_count INT DEFAULT 3,
+    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
+
+    INDEX idx_etl_jobs_active (is_active)
+);
+```
+
+### etl_job_runs - Execution tracking
+
+```sql
+CREATE TABLE etl_job_runs (
+    run_id BIGINT PRIMARY KEY AUTO_INCREMENT,
+    job_id VARCHAR(100) NOT NULL,
+    run_status VARCHAR(50) NOT NULL,        -- 'running', 'success', 'failed', 'cancelled'
+    started_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
+    finished_at TIMESTAMP NULL,
+    duration_seconds INT,
+    -- Metrics
+    rows_read BIGINT DEFAULT 0,
+    rows_written BIGINT DEFAULT 0,
+    rows_updated BIGINT DEFAULT 0,
+    rows_deleted BIGINT DEFAULT 0,
+    -- Watermarks
+    watermark_start TIMESTAMP NULL,
+    watermark_end TIMESTAMP NULL,
+    -- Error info
+    error_message TEXT,
+    error_details JSON,
+    -- Metadata
+    triggered_by VARCHAR(100),              -- 'scheduler', 'manual', 'api'
+
+    INDEX idx_etl_runs_job (job_id, started_at DESC),
+    INDEX idx_etl_runs_status (run_status, started_at),
+    FOREIGN KEY (job_id) REFERENCES etl_jobs(job_id)
+);
+```
+
+### etl_watermarks - Checkpoint tracking
+
+```sql
+CREATE TABLE etl_watermarks (
+    job_id VARCHAR(100) NOT NULL,
+    source_table VARCHAR(255) NOT NULL,
+    watermark_column VARCHAR(100) NOT NULL,  -- 'updated_at', 'id'
+    watermark_type VARCHAR(50) NOT NULL,     -- 'timestamp', 'integer'
+    last_value_timestamp TIMESTAMP NULL,
+    last_value_integer BIGINT NULL,
+    last_successful_run_id BIGINT,
+    last_updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
+
+    PRIMARY KEY (job_id, source_table),
+    FOREIGN KEY (job_id) REFERENCES etl_jobs(job_id)
+);
+```
+
+---
+
+## 3. Incremental Load Patterns
+
+### Timestamp-based (most common)
+
+```sql
+-- Query incremental data from source
+SELECT * FROM orders
+WHERE updated_at > :last_watermark
+  AND updated_at <= :current_watermark
+ORDER BY updated_at;
+
+-- Update watermark AFTER successful load
+UPDATE etl_watermarks
+SET last_value_timestamp = :current_watermark,
+    last_successful_run_id = :run_id,
+    last_updated_at = CURRENT_TIMESTAMP
+WHERE job_id = :job_id AND source_table = 'orders';
+```
+
+### ID-based (for append-only tables)
+
+```sql
+-- Query incremental data
+SELECT * FROM events
+WHERE id > :last_id
+ORDER BY id
+LIMIT 10000;
+
+-- Update watermark
+UPDATE etl_watermarks
+SET last_value_integer = :max_id_loaded,
+    last_successful_run_id = :run_id
+WHERE job_id = :job_id;
+```
+
+### Combined (timestamp + ID)
+
+```sql
+SELECT * FROM orders
+WHERE (updated_at > :last_timestamp)
+   OR (updated_at = :last_timestamp AND id > :last_id)
+ORDER BY updated_at, id
+LIMIT 10000;
+```
+
+---
+
+## 4. Example Queries
+
+### Get latest job status
+
+```sql
+SELECT
+    j.job_id, j.job_name, j.load_type,
+    r.run_status, r.started_at, r.finished_at,
+    r.rows_written, r.error_message
+FROM etl_jobs j
+LEFT JOIN (
+    SELECT * FROM etl_job_runs r1
+    WHERE started_at = (
+        SELECT MAX(started_at) FROM etl_job_runs r2
+        WHERE r2.job_id = r1.job_id
+    )
+) r ON j.job_id = r.job_id
+WHERE j.is_active = TRUE;
+```
+
+### Jobs failed in last 24 hours
+
+```sql
+SELECT j.job_name, r.run_id, r.started_at, r.error_message
+FROM etl_job_runs r
+JOIN etl_jobs j ON r.job_id = j.job_id
+WHERE r.run_status = 'failed'
+  AND r.started_at >= NOW() - INTERVAL 24 HOUR;
+```
+
+---
+
+## 5. Best Practices
+
+### Watermark management
+- Commit watermark **AFTER** data has been successfully loaded
+- Use transactions to ensure consistency
+- Store both watermark_start and watermark_end for each run
+
+### Error handling
+- Log enough context for debugging: source_record_id, source_data, error_stack
+- Classify errors: retryable vs non-retryable
+- Set up alerts for critical failures
+
+### Retention policy
+- Keep detailed logs: 30-90 days
+- Keep summary: 1-2 years
+- Keep watermarks: indefinitely
+
+---
+
+## Checklist
+
+- [ ] Source tables have `updated_at` with index
+- [ ] ETL control tables created
+- [ ] Watermark committed AFTER successful load
+- [ ] Sufficient logging for debugging: rows, timestamps, errors
+- [ ] Retention policy for ETL logs (30-90 days)
+- [ ] Alerts for failed jobs