cokit-cli 1.0.9 → 1.2.0

package/skills/databases/analytics.md

@@ -0,0 +1,198 @@
+# Analytics (OLAP) Rules
+
+> **Note:** Core naming conventions, workflow, and checklist are in `SKILL.md` or `db-design.md` (always loaded).
+
+Guidelines for designing schemas for statistics and reporting tables.
+
+---
+
+## General Principles
+
+- **Separate** from transactional tables - don't mix analytics logic into business tables
+- When heavy analytics queries/aggregations repeat → create separate tables
+- Use **Star Schema**: Fact tables at center, Dimension tables around
+
+---
+
+## Design Process
+
+### 1. Analyze Statistics Requirements
+
+Ask user to clarify:
+- **Analysis dimensions**: by date, by customer, by product, by channel, by region?
+- **Granularity**: per order, per item, per day, per month?
+- **Metrics**: order_count, revenue, margin, conversion_rate, avg_order_value?
+
+### 2. Define Fact Granularity
+
+**Important**: What does 1 row in fact table represent?
+
+| Fact Table | Granularity | Use case |
+|------------|-------------|----------|
+| `fact_orders` | 1 row = 1 order | Statistics by order |
+| `fact_order_items` | 1 row = 1 order item | Statistics by product |
+| `fact_daily_sales` | 1 row = 1 day + store | Daily summary |
+
+### 3. Identify Required Dimensions
+
+Create separate dim table when:
+- Reused in multiple places
+- Has many descriptive attributes
+- Subject to slow changes (Slowly Changing Dimension)
+
+---
+
+## Fact Tables
+
+### Fact table structure
+
+```sql
+CREATE TABLE fact_orders (
+    fact_id BIGINT PRIMARY KEY AUTO_INCREMENT,
+    -- Dimension keys
+    date_key INT NOT NULL,              -- FK to dim_date
+    customer_key BIGINT NOT NULL,       -- FK to dim_customer
+    store_key INT,
+    channel_key INT,
+    -- Degenerate dimensions (no separate dim needed)
+    order_id BIGINT NOT NULL,
+    order_number VARCHAR(50),
+    -- Measures
+    item_count INT NOT NULL,
+    gross_amount DECIMAL(18,2) NOT NULL,
+    discount_amount DECIMAL(18,2) DEFAULT 0,
+    net_amount DECIMAL(18,2) NOT NULL,
+
+    INDEX idx_fact_orders_date (date_key),
+    INDEX idx_fact_orders_customer (customer_key),
+    INDEX idx_fact_orders_date_store (date_key, store_key)
+);
+```
+
+---
+
+## Dimension Tables
+
+### dim_date (required for every analytics schema)
+
+```sql
+CREATE TABLE dim_date (
+    date_key INT PRIMARY KEY,           -- Format: YYYYMMDD (20241215)
+    full_date DATE NOT NULL,
+    year INT NOT NULL,
+    quarter INT NOT NULL,               -- 1-4
+    month INT NOT NULL,                 -- 1-12
+    month_name VARCHAR(20),             -- 'January', 'February'
+    week_of_year INT NOT NULL,
+    day_of_month INT NOT NULL,
+    day_of_week INT NOT NULL,           -- 1=Monday, 7=Sunday
+    day_name VARCHAR(20),
+    is_weekend BOOLEAN NOT NULL,
+    is_holiday BOOLEAN DEFAULT FALSE,
+
+    UNIQUE (full_date)
+);
+-- Pre-populate for multiple years (2020-2030)
+```
+
+### dim_customer
+
+```sql
+CREATE TABLE dim_customer (
+    customer_key BIGINT PRIMARY KEY AUTO_INCREMENT,  -- Surrogate key
+    customer_id BIGINT NOT NULL,                     -- Natural key from users
+    customer_name VARCHAR(255),
+    email VARCHAR(255),
+    segment VARCHAR(50),                -- 'VIP', 'Regular', 'New'
+    city VARCHAR(100),
+    region VARCHAR(100),
+    first_order_date DATE,
+    -- SCD Type 2 columns (if history needed)
+    effective_from DATE NOT NULL,
+    effective_to DATE,
+    is_current BOOLEAN DEFAULT TRUE,
+
+    INDEX idx_dim_customer_id (customer_id),
+    INDEX idx_dim_customer_current (is_current, customer_id)
+);
+```
+
+---
+
+## Summary Tables (Pre-aggregated)
+
+When pre-aggregation needed for dashboard performance:
+
+```sql
+CREATE TABLE summary_daily_sales (
+    id BIGINT PRIMARY KEY AUTO_INCREMENT,
+    date_key INT NOT NULL,
+    store_key INT,
+    channel_key INT,
+    -- Pre-aggregated measures
+    order_count INT NOT NULL,
+    item_count INT NOT NULL,
+    gross_revenue DECIMAL(18,2) NOT NULL,
+    net_revenue DECIMAL(18,2) NOT NULL,
+    unique_customers INT NOT NULL,
+    avg_order_value DECIMAL(18,2),
+
+    UNIQUE (date_key, store_key, channel_key),
+    INDEX idx_summary_date (date_key)
+);
+```
+
+---
+
+## Slowly Changing Dimensions (SCD)
+
+### Type 1 - Overwrite
+Overwrite old value, no history kept:
+```sql
+UPDATE dim_customer SET segment = 'VIP' WHERE customer_id = 123;
+```
+
+### Type 2 - Add new row (Recommended when history needed)
+```sql
+-- 1. Close old row
+UPDATE dim_customer
+SET effective_to = CURRENT_DATE - 1, is_current = FALSE
+WHERE customer_id = 123 AND is_current = TRUE;
+
+-- 2. Add new row
+INSERT INTO dim_customer (customer_id, segment, effective_from, is_current)
+VALUES (123, 'VIP', CURRENT_DATE, TRUE);
+```
+
+---
+
+## Indexing for Analytics
+
+### Fact tables
+- Index FKs to dimensions: `date_key`, `customer_key`, `product_key`
+- Composite index based on query patterns: `INDEX (date_key, store_key)`
+
+### Dimension tables
+- PK: surrogate key
+- Index natural key: `customer_id`, `product_id`
+- Index for SCD: `(is_current, customer_id)`
+
+---
+
+## Naming Convention
+
+- Fact tables: `fact_*` or `fct_*`
+- Dimension tables: `dim_*`
+- Summary tables: `summary_*` or `agg_*`
+
+---
+
+## Checklist
+
+- [ ] Granularity defined for each fact table
+- [ ] dim_date exists or created (pre-populate multiple years)
+- [ ] Surrogate keys for dimensions
+- [ ] Index FKs in fact tables
+- [ ] SCD strategy for changing dimensions (Type 1 or Type 2)
+- [ ] Naming: `fact_*`, `dim_*`, `summary_*`
+- [ ] Refresh strategy: see [incremental-etl.md](incremental-etl.md)

package/skills/databases/db-design.md

@@ -0,0 +1,188 @@
+---
+name: db-design
+description: "Activate when user requests: Database/table design for transactional (OLTP), analytics (OLAP), create or extend schema, design fact/dimension tables, analyze/review CSV/JSON/SQL files to create tables, or need advice on data storage structure."
+---
+
+# SKILL: Database Schema Design
+
+## Execution Process
+
+### Step 1: Analyze Context (MANDATORY)
+
+Before ANY design work, AI MUST check:
+
+- [ ] Database stack identified? (MySQL / PostgreSQL / BigQuery / SQLite / D1)
+- [ ] Existing schema explored via tools/MCP?
+- [ ] Use case clear? (transactional vs analytics vs ETL)
+
+**If ANY checkbox is unclear → STOP and ask user before proceeding.**
+
+### Step 2: Gather Requirements
+
+**MUST ask if missing:**
+
+| Missing Info | Required Question |
+|--------------|-------------------|
+| Entities unclear | "What are the main entities involved in this feature?" |
+| Query patterns unknown | "What queries/reports will run most often?" |
+| Granularity unclear | "What level of detail needed? (per order, per item, per day)" |
+| Constraints unknown | "Any uniqueness, retention, or multi-tenant requirements?" |
+| Relationships unclear | "How do these entities relate to each other?" |
+
+**DO NOT assume or proceed without answers. DO NOT design schema without understanding the business context.**
+
+### Step 3: Load Documentation by Intent
+
+| Intent | Keywords | Load file |
+|--------|----------|-----------|
+| **OLTP** | "business tables", "transactional", "CRUD", "orders", "users" | [transactional.md](transactional.md) |
+| **OLAP** | "analytics", "statistics", "reports", "fact", "dimension" | [analytics.md](analytics.md) |
+| **ETL** | "incremental", "load", "sync", "watermark", "etl" | [incremental-etl.md](incremental-etl.md) |
+
+**Database stacks:** [MySQL](stacks/mysql.md) · [PostgreSQL](stacks/postgres.md) · [BigQuery](stacks/bigquery.md) · [SQLite](stacks/sqlite.md) · [D1 Cloudflare](stacks/d1_cloudflare.md)
+
+### Step 4: Design & Propose
+
+Only after Steps 1-3 complete:
+- Propose DDL with reasoning
+- **Wait for user approval before execution**
+- Never auto-execute DDL
+
+### Step 5: Verify Checklist
+
+Before delivering final DDL, verify the **Final Checklist** at the end of this file.
+
+---
+
+## Scope & Objectives
+
+Design or extend database schemas ensuring:
+
+- Consistent, understandable, maintainable data structures
+- Good performance with appropriate indexes
+- Design that fits existing tables (avoid duplication, avoid breaking logic)
+
+---
+
+## Naming Conventions
+
+### Table naming
+
+- Use **snake_case**, English, plural for main entities:
+  - `users`, `orders`, `order_items`, `product_variants`
+- Junction/mapping tables:
+  - `role_permissions`, `user_roles`, `product_categories`
+- Fact & dimension tables (analytics):
+  - Fact: `fact_orders`, `fact_order_items`, `fact_pageviews`
+  - Dimension: `dim_date`, `dim_customer`, `dim_product`, `dim_store`
+- Avoid unclear abbreviations:
+  - ❌ `usr`, `ord`, `inv_tx`
+  - ✅ `users`, `orders`, `inventory_transactions`
+
+### Column naming
+
+- Use **snake_case**, clear meaning, consistent across tables
+- **Primary key**: `id` (main PK of table)
+- **Foreign key**: `xxx_id` pointing to table `xxx`:
+  - `user_id` → `users.id`
+  - `order_id` → `orders.id`
+- **Audit & lifecycle**:
+  - `created_at`, `updated_at`, `deleted_at` (soft delete)
+- **Boolean flags**:
+  - `is_active`, `is_deleted`, `is_primary`, `is_verified`
+- **Quantities & prices**:
+  - `quantity`, `unit_price`, `total_price`, `discount_amount`, `tax_amount`
+- **Time ranges**:
+  - `start_at`, `end_at` or `valid_from`, `valid_to` (pick one convention and use consistently)
+- **Enum/state**:
+  - `status` (combine with ENUM / CHECK constraint)
+
+---
+
+## Workflow
+
+### 1. Understand business requirements
+
+- Summarize requirements in your own words
+- If missing context, ask:
+  - What entities are involved?
+  - What is the main business flow?
+  - What queries/reports are commonly used?
+
+### 2. Explore existing schema (MANDATORY before proposing)
+
+- Use tools/MCP to list tables and describe structure
+- For DBs with comments (MySQL/PostgreSQL/BigQuery): read comments directly
+- For SQLite/D1: query `metadata_tables` and `metadata_columns` tables
+
+### 3. Analyze and propose draft
+
+- Which tables already partially cover requirements?
+- Which tables need additional columns/indexes?
+- Which tables are completely missing (need to create)?
+
+### 4. Confirm with user before finalizing design
+
+- Required granularity
+- Metrics to calculate
+- Important business constraints (uniqueness, retention, privacy)
+
+### 5. Design detailed schema
+
+- Provide proposed DDL - **do NOT auto-execute**
+- Wait for user approval before execution
+
+---
+
+## Processing Data Files (CSV/JSON/SQL)
+
+When user provides files:
+
+1. **Read and identify structure**: headers, data types, relationships
+2. **Large files**: write Python script to parse and analyze
+3. **SQL files**: review naming, indexes, constraints, propose improvements
+4. **Output**: Proposed DDL with reasoning
+
+---
+
+## Output Format
+
+When delivering design results:
+
+1. **Summary**: list of new tables / tables to modify
+2. **Complete DDL SQL**: columns, types, PK, FK, indexes, comments
+3. **Reasoning**: explain why this structure/index was chosen, tied to actual queries
+
+---
+
+## Final Checklist
+
+Before delivering final DDL:
+
+- [ ] Explored existing schema via tools/MCP
+- [ ] No duplicate tables with existing schema
+- [ ] All FKs have indexes
+- [ ] Composite indexes match main query patterns
+- [ ] Naming convention consistent with existing schema
+- [ ] Added comments/descriptions for tables and columns
+- [ ] For SQLite/D1: prepared INSERTs for metadata tables
+- [ ] Explained reasoning for important design decisions
+
+---
+
+## File Structure
+
+```
+$HOME/.copilot/skills/databases/
+├── SKILL.md                  # Skill description (entry + execution process + checklist)
+├── db-design.md              # This file 
+├── transactional.md          # OLTP design rules
+├── analytics.md              # OLAP / Star Schema rules
+├── incremental-etl.md        # ETL & watermark patterns
+└── stacks/
+    ├── mysql.md              # MySQL-specific rules
+    ├── postgres.md           # PostgreSQL-specific rules
+    ├── bigquery.md           # BigQuery-specific rules
+    ├── sqlite.md             # SQLite-specific rules
+    └── d1_cloudflare.md      # D1 Cloudflare-specific rules
+```

package/skills/databases/incremental-etl.md

@@ -0,0 +1,213 @@
+# Incremental Loading & ETL Logs
+
+> **Note:** Core naming conventions, workflow, and checklist are in `SKILL.md` or `db-design.md` (always loaded).
+
+Guidelines for designing schemas to support periodic data loading with tracking and control.
+
+---
+
+## 1. Source Tables Requirements
+
+All source tables that need to support incremental loading must have:
+
+```sql
+-- Required audit columns
+created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
+updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
+
+-- Index for incremental query
+CREATE INDEX idx_tablename_updated_at ON tablename(updated_at);
+```
+
+### Soft delete support
+
+If source has soft delete, need to track deleted records too:
+
+```sql
+deleted_at TIMESTAMP NULL
+
+-- Index to get all modified records (update or delete)
+CREATE INDEX idx_tablename_modified ON tablename(updated_at);
+```
+
+---
+
+## 2. ETL Control Tables
+
+### etl_jobs - Job definitions
+
+```sql
+CREATE TABLE etl_jobs (
+    job_id VARCHAR(100) PRIMARY KEY,
+    job_name VARCHAR(255) NOT NULL,
+    description TEXT,
+    source_table VARCHAR(255),
+    target_table VARCHAR(255),
+    load_type VARCHAR(50) NOT NULL,         -- 'full', 'incremental', 'cdc'
+    schedule VARCHAR(100),                   -- Cron: '0 */1 * * *'
+    is_active BOOLEAN DEFAULT TRUE,
+    timeout_seconds INT DEFAULT 3600,
+    retry_count INT DEFAULT 3,
+    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
+
+    INDEX idx_etl_jobs_active (is_active)
+);
+```
+
+### etl_job_runs - Execution tracking
+
+```sql
+CREATE TABLE etl_job_runs (
+    run_id BIGINT PRIMARY KEY AUTO_INCREMENT,
+    job_id VARCHAR(100) NOT NULL,
+    run_status VARCHAR(50) NOT NULL,        -- 'running', 'success', 'failed', 'cancelled'
+    started_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
+    finished_at TIMESTAMP NULL,
+    duration_seconds INT,
+    -- Metrics
+    rows_read BIGINT DEFAULT 0,
+    rows_written BIGINT DEFAULT 0,
+    rows_updated BIGINT DEFAULT 0,
+    rows_deleted BIGINT DEFAULT 0,
+    -- Watermarks
+    watermark_start TIMESTAMP NULL,
+    watermark_end TIMESTAMP NULL,
+    -- Error info
+    error_message TEXT,
+    error_details JSON,
+    -- Metadata
+    triggered_by VARCHAR(100),              -- 'scheduler', 'manual', 'api'
+
+    INDEX idx_etl_runs_job (job_id, started_at DESC),
+    INDEX idx_etl_runs_status (run_status, started_at),
+    FOREIGN KEY (job_id) REFERENCES etl_jobs(job_id)
+);
+```
+
+### etl_watermarks - Checkpoint tracking
+
+```sql
+CREATE TABLE etl_watermarks (
+    job_id VARCHAR(100) NOT NULL,
+    source_table VARCHAR(255) NOT NULL,
+    watermark_column VARCHAR(100) NOT NULL,  -- 'updated_at', 'id'
+    watermark_type VARCHAR(50) NOT NULL,     -- 'timestamp', 'integer'
+    last_value_timestamp TIMESTAMP NULL,
+    last_value_integer BIGINT NULL,
+    last_successful_run_id BIGINT,
+    last_updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
+
+    PRIMARY KEY (job_id, source_table),
+    FOREIGN KEY (job_id) REFERENCES etl_jobs(job_id)
+);
+```
+
+---
+
+## 3. Incremental Load Patterns
+
+### Timestamp-based (most common)
+
+```sql
+-- Query incremental data from source
+SELECT * FROM orders
+WHERE updated_at > :last_watermark
+  AND updated_at <= :current_watermark
+ORDER BY updated_at;
+
+-- Update watermark AFTER successful load
+UPDATE etl_watermarks
+SET last_value_timestamp = :current_watermark,
+    last_successful_run_id = :run_id,
+    last_updated_at = CURRENT_TIMESTAMP
+WHERE job_id = :job_id AND source_table = 'orders';
+```
+
+### ID-based (for append-only tables)
+
+```sql
+-- Query incremental data
+SELECT * FROM events
+WHERE id > :last_id
+ORDER BY id
+LIMIT 10000;
+
+-- Update watermark
+UPDATE etl_watermarks
+SET last_value_integer = :max_id_loaded,
+    last_successful_run_id = :run_id
+WHERE job_id = :job_id;
+```
+
+### Combined (timestamp + ID)
+
+```sql
+SELECT * FROM orders
+WHERE (updated_at > :last_timestamp)
+   OR (updated_at = :last_timestamp AND id > :last_id)
+ORDER BY updated_at, id
+LIMIT 10000;
+```
+
+---
+
+## 4. Example Queries
+
+### Get latest job status
+
+```sql
+SELECT
+    j.job_id, j.job_name, j.load_type,
+    r.run_status, r.started_at, r.finished_at,
+    r.rows_written, r.error_message
+FROM etl_jobs j
+LEFT JOIN (
+    SELECT * FROM etl_job_runs r1
+    WHERE started_at = (
+        SELECT MAX(started_at) FROM etl_job_runs r2
+        WHERE r2.job_id = r1.job_id
+    )
+) r ON j.job_id = r.job_id
+WHERE j.is_active = TRUE;
+```
+
+### Jobs failed in last 24 hours
+
+```sql
+SELECT j.job_name, r.run_id, r.started_at, r.error_message
+FROM etl_job_runs r
+JOIN etl_jobs j ON r.job_id = j.job_id
+WHERE r.run_status = 'failed'
+  AND r.started_at >= NOW() - INTERVAL 24 HOUR;
+```
+
+---
+
+## 5. Best Practices
+
+### Watermark management
+- Commit watermark **AFTER** data has been successfully loaded
+- Use transactions to ensure consistency
+- Store both watermark_start and watermark_end for each run
+
+### Error handling
+- Log enough context for debugging: source_record_id, source_data, error_stack
+- Classify errors: retryable vs non-retryable
+- Set up alerts for critical failures
+
+### Retention policy
+- Keep detailed logs: 30-90 days
+- Keep summary: 1-2 years
+- Keep watermarks: indefinitely
+
+---
+
+## Checklist
+
+- [ ] Source tables have `updated_at` with index
+- [ ] ETL control tables created
+- [ ] Watermark committed AFTER successful load
+- [ ] Sufficient logging for debugging: rows, timestamps, errors
+- [ ] Retention policy for ETL logs (30-90 days)
+- [ ] Alerts for failed jobs