@hienlh/ppm 0.9.81 → 0.9.83

package/.opencode/skills/databases/db-design.md

@@ -1,188 +0,0 @@
----
-name: db-design
-description: "Activate when user requests: Database/table design for transactional (OLTP), analytics (OLAP), create or extend schema, design fact/dimension tables, analyze/review CSV/JSON/SQL files to create tables, or need advice on data storage structure."
----
-
-# SKILL: Database Schema Design
-
-## Execution Process
-
-### Step 1: Analyze Context (MANDATORY)
-
-Before ANY design work, AI MUST check:
-
-- [ ] Database stack identified? (MySQL / PostgreSQL / BigQuery / SQLite / D1)
-- [ ] Existing schema explored via tools/MCP?
-- [ ] Use case clear? (transactional vs analytics vs ETL)
-
-**If ANY checkbox is unclear → STOP and ask user before proceeding.**
-
-### Step 2: Gather Requirements
-
-**MUST ask if missing:**
-
-| Missing Info | Required Question |
-|--------------|-------------------|
-| Entities unclear | "What are the main entities involved in this feature?" |
-| Query patterns unknown | "What queries/reports will run most often?" |
-| Granularity unclear | "What level of detail needed? (per order, per item, per day)" |
-| Constraints unknown | "Any uniqueness, retention, or multi-tenant requirements?" |
-| Relationships unclear | "How do these entities relate to each other?" |
-
-**DO NOT assume or proceed without answers. DO NOT design schema without understanding the business context.**
-
-### Step 3: Load Documentation by Intent
-
-| Intent | Keywords | Load file |
-|--------|----------|-----------|
-| **OLTP** | "business tables", "transactional", "CRUD", "orders", "users" | [transactional.md](transactional.md) |
-| **OLAP** | "analytics", "statistics", "reports", "fact", "dimension" | [analytics.md](analytics.md) |
-| **ETL** | "incremental", "load", "sync", "watermark", "etl" | [incremental-etl.md](incremental-etl.md) |
-
-**Database stacks:** [MySQL](stacks/mysql.md) · [PostgreSQL](stacks/postgres.md) · [BigQuery](stacks/bigquery.md) · [SQLite](stacks/sqlite.md) · [D1 Cloudflare](stacks/d1_cloudflare.md)
-
-### Step 4: Design & Propose
-
-Only after Steps 1-3 complete:
-- Propose DDL with reasoning
-- **Wait for user approval before execution**
-- Never auto-execute DDL
-
-### Step 5: Verify Checklist
-
-Before delivering final DDL, verify the **Final Checklist** at the end of this file.
-
----
-
-## Scope & Objectives
-
-Design or extend database schemas ensuring:
-
-- Consistent, understandable, maintainable data structures
-- Good performance with appropriate indexes
-- Design that fits existing tables (avoid duplication, avoid breaking logic)
-
----
-
-## Naming Conventions
-
-### Table naming
-
-- Use **snake_case**, English, plural for main entities:
-  - `users`, `orders`, `order_items`, `product_variants`
-- Junction/mapping tables:
-  - `role_permissions`, `user_roles`, `product_categories`
-- Fact & dimension tables (analytics):
-  - Fact: `fact_orders`, `fact_order_items`, `fact_pageviews`
-  - Dimension: `dim_date`, `dim_customer`, `dim_product`, `dim_store`
-- Avoid unclear abbreviations:
-  - ❌ `usr`, `ord`, `inv_tx`
-  - ✅ `users`, `orders`, `inventory_transactions`
-
-### Column naming
-
-- Use **snake_case**, clear meaning, consistent across tables
-- **Primary key**: `id` (main PK of table)
-- **Foreign key**: `xxx_id` pointing to table `xxx`:
-  - `user_id` → `users.id`
-  - `order_id` → `orders.id`
-- **Audit & lifecycle**:
-  - `created_at`, `updated_at`, `deleted_at` (soft delete)
-- **Boolean flags**:
-  - `is_active`, `is_deleted`, `is_primary`, `is_verified`
-- **Quantities & prices**:
-  - `quantity`, `unit_price`, `total_price`, `discount_amount`, `tax_amount`
-- **Time ranges**:
-  - `start_at`, `end_at` or `valid_from`, `valid_to` (pick one convention and use consistently)
-- **Enum/state**:
-  - `status` (combine with ENUM / CHECK constraint)
-
----
-
-## Workflow
-
-### 1. Understand business requirements
-
-- Summarize requirements in your own words
-- If missing context, ask:
-  - What entities are involved?
-  - What is the main business flow?
-  - What queries/reports are commonly used?
-
-### 2. Explore existing schema (MANDATORY before proposing)
-
-- Use tools/MCP to list tables and describe structure
-- For DBs with comments (MySQL/PostgreSQL/BigQuery): read comments directly
-- For SQLite/D1: query `metadata_tables` and `metadata_columns` tables
-
-### 3. Analyze and propose draft
-
-- Which tables already partially cover requirements?
-- Which tables need additional columns/indexes?
-- Which tables are completely missing (need to create)?
-
-### 4. Confirm with user before finalizing design
-
-- Required granularity
-- Metrics to calculate
-- Important business constraints (uniqueness, retention, privacy)
-
-### 5. Design detailed schema
-
-- Provide proposed DDL - **do NOT auto-execute**
-- Wait for user approval before execution
-
----
-
-## Processing Data Files (CSV/JSON/SQL)
-
-When user provides files:
-
-1. **Read and identify structure**: headers, data types, relationships
-2. **Large files**: write Python script to parse and analyze
-3. **SQL files**: review naming, indexes, constraints, propose improvements
-4. **Output**: Proposed DDL with reasoning
-
----
-
-## Output Format
-
-When delivering design results:
-
-1. **Summary**: list of new tables / tables to modify
-2. **Complete DDL SQL**: columns, types, PK, FK, indexes, comments
-3. **Reasoning**: explain why this structure/index was chosen, tied to actual queries
-
----
-
-## Final Checklist
-
-Before delivering final DDL:
-
-- [ ] Explored existing schema via tools/MCP
-- [ ] No duplicate tables with existing schema
-- [ ] All FKs have indexes
-- [ ] Composite indexes match main query patterns
-- [ ] Naming convention consistent with existing schema
-- [ ] Added comments/descriptions for tables and columns
-- [ ] For SQLite/D1: prepared INSERTs for metadata tables
-- [ ] Explained reasoning for important design decisions
-
----
-
-## File Structure
-
-```
-.opencode/skills/databases/
-├── SKILL.md                  # Skill description (entry + execution process + checklist)
-├── db-design.md              # This file 
-├── transactional.md          # OLTP design rules
-├── analytics.md              # OLAP / Star Schema rules
-├── incremental-etl.md        # ETL & watermark patterns
-└── stacks/
-    ├── mysql.md              # MySQL-specific rules
-    ├── postgres.md           # PostgreSQL-specific rules
-    ├── bigquery.md           # BigQuery-specific rules
-    ├── sqlite.md             # SQLite-specific rules
-    └── d1_cloudflare.md      # D1 Cloudflare-specific rules
-```

package/.opencode/skills/databases/incremental-etl.md

@@ -1,213 +0,0 @@
-# Incremental Loading & ETL Logs
-
-> **Note:** Core naming conventions, workflow, and checklist are in `SKILL.md` or `db-design.md` (always loaded).
-
-Guidelines for designing schemas to support periodic data loading with tracking and control.
-
----
-
-## 1. Source Tables Requirements
-
-All source tables that need to support incremental loading must have:
-
-```sql
--- Required audit columns
-created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
-updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
-
--- Index for incremental query
-CREATE INDEX idx_tablename_updated_at ON tablename(updated_at);
-```
-
-### Soft delete support
-
-If source has soft delete, need to track deleted records too:
-
-```sql
-deleted_at TIMESTAMP NULL
-
--- Index to get all modified records (update or delete)
-CREATE INDEX idx_tablename_modified ON tablename(updated_at);
-```
-
----
-
-## 2. ETL Control Tables
-
-### etl_jobs - Job definitions
-
-```sql
-CREATE TABLE etl_jobs (
-    job_id VARCHAR(100) PRIMARY KEY,
-    job_name VARCHAR(255) NOT NULL,
-    description TEXT,
-    source_table VARCHAR(255),
-    target_table VARCHAR(255),
-    load_type VARCHAR(50) NOT NULL,         -- 'full', 'incremental', 'cdc'
-    schedule VARCHAR(100),                   -- Cron: '0 */1 * * *'
-    is_active BOOLEAN DEFAULT TRUE,
-    timeout_seconds INT DEFAULT 3600,
-    retry_count INT DEFAULT 3,
-    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
-    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
-
-    INDEX idx_etl_jobs_active (is_active)
-);
-```
-
-### etl_job_runs - Execution tracking
-
-```sql
-CREATE TABLE etl_job_runs (
-    run_id BIGINT PRIMARY KEY AUTO_INCREMENT,
-    job_id VARCHAR(100) NOT NULL,
-    run_status VARCHAR(50) NOT NULL,        -- 'running', 'success', 'failed', 'cancelled'
-    started_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
-    finished_at TIMESTAMP NULL,
-    duration_seconds INT,
-    -- Metrics
-    rows_read BIGINT DEFAULT 0,
-    rows_written BIGINT DEFAULT 0,
-    rows_updated BIGINT DEFAULT 0,
-    rows_deleted BIGINT DEFAULT 0,
-    -- Watermarks
-    watermark_start TIMESTAMP NULL,
-    watermark_end TIMESTAMP NULL,
-    -- Error info
-    error_message TEXT,
-    error_details JSON,
-    -- Metadata
-    triggered_by VARCHAR(100),              -- 'scheduler', 'manual', 'api'
-
-    INDEX idx_etl_runs_job (job_id, started_at DESC),
-    INDEX idx_etl_runs_status (run_status, started_at),
-    FOREIGN KEY (job_id) REFERENCES etl_jobs(job_id)
-);
-```
-
-### etl_watermarks - Checkpoint tracking
-
-```sql
-CREATE TABLE etl_watermarks (
-    job_id VARCHAR(100) NOT NULL,
-    source_table VARCHAR(255) NOT NULL,
-    watermark_column VARCHAR(100) NOT NULL,  -- 'updated_at', 'id'
-    watermark_type VARCHAR(50) NOT NULL,     -- 'timestamp', 'integer'
-    last_value_timestamp TIMESTAMP NULL,
-    last_value_integer BIGINT NULL,
-    last_successful_run_id BIGINT,
-    last_updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
-
-    PRIMARY KEY (job_id, source_table),
-    FOREIGN KEY (job_id) REFERENCES etl_jobs(job_id)
-);
-```
-
----
-
-## 3. Incremental Load Patterns
-
-### Timestamp-based (most common)
-
-```sql
--- Query incremental data from source
-SELECT * FROM orders
-WHERE updated_at > :last_watermark
-  AND updated_at <= :current_watermark
-ORDER BY updated_at;
-
--- Update watermark AFTER successful load
-UPDATE etl_watermarks
-SET last_value_timestamp = :current_watermark,
-    last_successful_run_id = :run_id,
-    last_updated_at = CURRENT_TIMESTAMP
-WHERE job_id = :job_id AND source_table = 'orders';
-```
-
-### ID-based (for append-only tables)
-
-```sql
--- Query incremental data
-SELECT * FROM events
-WHERE id > :last_id
-ORDER BY id
-LIMIT 10000;
-
--- Update watermark
-UPDATE etl_watermarks
-SET last_value_integer = :max_id_loaded,
-    last_successful_run_id = :run_id
-WHERE job_id = :job_id;
-```
-
-### Combined (timestamp + ID)
-
-```sql
-SELECT * FROM orders
-WHERE (updated_at > :last_timestamp)
-   OR (updated_at = :last_timestamp AND id > :last_id)
-ORDER BY updated_at, id
-LIMIT 10000;
-```
-
----
-
-## 4. Example Queries
-
-### Get latest job status
-
-```sql
-SELECT
-    j.job_id, j.job_name, j.load_type,
-    r.run_status, r.started_at, r.finished_at,
-    r.rows_written, r.error_message
-FROM etl_jobs j
-LEFT JOIN (
-    SELECT * FROM etl_job_runs r1
-    WHERE started_at = (
-        SELECT MAX(started_at) FROM etl_job_runs r2
-        WHERE r2.job_id = r1.job_id
-    )
-) r ON j.job_id = r.job_id
-WHERE j.is_active = TRUE;
-```
-
-### Jobs failed in last 24 hours
-
-```sql
-SELECT j.job_name, r.run_id, r.started_at, r.error_message
-FROM etl_job_runs r
-JOIN etl_jobs j ON r.job_id = j.job_id
-WHERE r.run_status = 'failed'
-  AND r.started_at >= NOW() - INTERVAL 24 HOUR;
-```
-
----
-
-## 5. Best Practices
-
-### Watermark management
-- Commit watermark **AFTER** data has been successfully loaded
-- Use transactions to ensure consistency
-- Store both watermark_start and watermark_end for each run
-
-### Error handling
-- Log enough context for debugging: source_record_id, source_data, error_stack
-- Classify errors: retryable vs non-retryable
-- Set up alerts for critical failures
-
-### Retention policy
-- Keep detailed logs: 30-90 days
-- Keep summary: 1-2 years
-- Keep watermarks: indefinitely
-
----
-
-## Checklist
-
-- [ ] Source tables have `updated_at` with index
-- [ ] ETL control tables created
-- [ ] Watermark committed AFTER successful load
-- [ ] Sufficient logging for debugging: rows, timestamps, errors
-- [ ] Retention policy for ETL logs (30-90 days)
-- [ ] Alerts for failed jobs