everything-gemini 1.1.1 → 1.8.0

package/skills/dbt-transformation/SKILL.md

@@ -0,0 +1,179 @@
+---
+name: dbt-transformation
+description: >-
+  Designs, develops, and tests modular SQL data transformations using dbt (data build tool), Jinja macros, semantic layer metrics, automated schema tests, and documentation generation.
+---
+
+# dbt Modular Transformation & Data Modeling
+
+This skill guides structuring production-grade ELT pipelines using dbt (Core & Cloud), enforcing modular layer architecture (staging, intermediate, marts), DRY Jinja macros, automated data testing, and incremental table materialization.
+
+---
+
+## ⚡ Core Patterns
+
+### 1. Multi-Layer Dimensional Modeling Architecture
+
+Organize transformations into clean, progressive DAG layers:
+
+```text
+models/
+├── staging/                     # 1:1 view on raw sources (renaming, casting)
+│   ├── stripe/
+│   │   ├── _stripe__sources.yml
+│   │   ├── _stripe__models.yml
+│   │   └── stg_stripe__payments.sql
+├── intermediate/                # Business logic joins & window aggregations
+│   └── finance/
+│       ├── _int_finance__models.yml
+│       └── int_payments_pivoted_to_orders.sql
+└── marts/                       # Dimensional star schema (facts & dimensions)
+    ├── core/
+    │   ├── fct_orders.sql
+    │   └── dim_customers.sql
+    └── marketing/
+        └── fct_campaign_conversions.sql
+```
+
+---
+
+### 2. Staging Layer Model with CTE Standards
+
+Clean raw source tables into normalized relational types:
+
+```sql
+-- models/staging/stripe/stg_stripe__payments.sql
+with source as (
+    select * from {{ source('stripe', 'payment') }}
+),
+
+renamed as (
+    select
+        id as payment_id,
+        orderid as order_id,
+        paymentmethod as payment_method,
+        status as payment_status,
+        -- Amount stored in cents; convert to dollars
+        round(amount / 100.0, 2) as amount_usd,
+        created as created_at
+    from source
+)
+
+select * from renamed
+```
+
+Declare tests and column descriptions in accompanying YAML:
+
+```yaml
+# models/staging/stripe/_stripe__models.yml
+version: 2
+
+models:
+  - name: stg_stripe__payments
+    description: "Normalized payments table from Stripe webhook ingest."
+    columns:
+      - name: payment_id
+        description: "Primary key for payment events."
+        tests:
+          - unique
+          - not_null
+      - name: payment_status
+        tests:
+          - accepted_values:
+              values: ['success', 'fail', 'pending']
+```
+
+---
+
+### 3. High-Performance Incremental Fact Table
+
+Process only newly ingested or updated records using `is_incremental()`:
+
+```sql
+-- models/marts/core/fct_orders.sql
+{{
+    config(
+        materialized='incremental',
+        unique_key='order_id',
+        incremental_strategy='merge',
+        on_schema_change='fail'
+    )
+}}
+
+with orders as (
+    select * from {{ ref('stg_jaffle_shop__orders') }}
+    {% if is_incremental() %}
+        -- Only query rows created or modified in the last 3 days for backfill resilience
+        where order_date >= (select dateadd('day', -3, max(order_date)) from {{ this }})
+    {% endif %}
+),
+
+payments as (
+    select * from {{ ref('stg_stripe__payments') }}
+),
+
+order_payments as (
+    select
+        order_id,
+        sum(case when payment_status = 'success' then amount_usd else 0 end) as amount_usd
+    from payments
+    group by 1
+),
+
+final as (
+    select
+        orders.order_id,
+        orders.customer_id,
+        orders.order_date,
+        orders.status as order_status,
+        coalesce(order_payments.amount_usd, 0) as total_amount_usd
+    from orders
+    left join order_payments on orders.order_id = order_payments.order_id
+)
+
+select * from final
+```
+
+---
+
+### 4. Custom Jinja Macro for Dynamic Pivoting
+
+Avoid repetitive boilerplate using parameterized Jinja macros:
+
+```sql
+-- macros/pivot_sum.sql
+{% macro pivot_sum(column_name, values_list, aggregate_col='amount_usd') %}
+    {% for val in values_list %}
+        sum(case when {{ column_name }} = '{{ val }}' then {{ aggregate_col }} else 0 end) as {{ val }}_amount
+        {% if not loop.last %},{% endif %}
+    {% endfor %}
+{% endmacro %}
+```
+
+Usage in intermediate model:
+
+```sql
+select
+    order_id,
+    {{ pivot_sum('payment_method', ['credit_card', 'coupon', 'bank_transfer', 'gift_card']) }}
+from {{ ref('stg_stripe__payments') }}
+group by 1
+```
+
+---
+
+## 🎯 Verification & Diagnostics
+
+1. **Compile SQL with Resolved Jinja**:
+   ```bash
+   dbt compile --select fct_orders
+   ```
+2. **Execute Targeted Subgraph**:
+   ```bash
+   dbt run --select +fct_orders       # Run fct_orders and all upstream dependencies
+   dbt test --select stg_stripe+      # Test staging models and downstream marts
+   ```
+3. **Build Documentation & DAG Visualizer**:
+   ```bash
+   dbt docs generate && dbt docs serve
+   ```

package/skills/debezium-cdc/SKILL.md

@@ -0,0 +1,166 @@
+---
+name: debezium-cdc
+description: >-
+  Change Data Capture (CDC) streaming architectures using Debezium, PostgreSQL logical decoding (pgoutput), MySQL binlog replication, Kafka Connect, and transactional outbox patterns. Use when capturing real-time database mutations, synchronizing search indices, or avoiding dual-write race conditions.
+---
+
+# Debezium Change Data Capture (CDC) Architecture
+
+A production runbook for implementing distributed **Change Data Capture (CDC)** pipelines using Debezium, PostgreSQL/MySQL WAL replication, Kafka Connect, and event-driven data streaming.
+
+---
+
+## 🎯 Core Architectural Principles
+
+1. **Log-Based CDC Over Polling**: Reads write-ahead logs (WAL / binlog) directly at the storage engine level, incurring minimal CPU overhead and zero query locks on primary tables.
+2. **Transactional Outbox Pattern**: Guarantees atomic database operations and event emission within a single ACID transaction without two-phase commit (2PC) or dual-write inconsistencies.
+3. **Strict Ordering Guarantees**: Partitioning by primary key ensures sequential ordering per record entity across consumer groups.
+4. **Idempotent Consumers**: Design downstream event processors with upsert semantics and revision monotonic version checks.
+
+---
+
+## 🏗️ Debezium PostgreSQL Connector Configuration
+
+### 1. Database Prerequisite (`postgresql.conf`)
+
+```ini
+# Enable logical replication
+wal_level = logical
+max_wal_senders = 10
+max_replication_slots = 10
+```
+
+### 2. Kafka Connect Connector Payload (`debezium-postgres.json`)
+
+```json
+{
+  "name": "postgres-order-cdc",
+  "config": {
+    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
+    "tasks.max": "1",
+    "plugin.name": "pgoutput",
+    "database.hostname": "postgres.internal",
+    "database.port": "5432",
+    "database.user": "debezium_cdc",
+    "database.password": "${file:/secrets/db-credentials.properties:password}",
+    "database.dbname": "production_app",
+    "database.server.name": "prod_cdc",
+    "table.include.list": "public.orders,public.outbox_events",
+    "slot.name": "debezium_orders_slot",
+    "publication.name": "dbz_orders_pub",
+    "publication.autocreate.mode": "filtered",
+    "decimal.handling.mode": "double",
+    "tombstones.on.delete": "true",
+    "heartbeat.interval.ms": "5000"
+  }
+}
+```
+
+Deploy to Kafka Connect cluster via REST API:
+
+```bash
+curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" \
+  http://kafka-connect.internal:8083/connectors/ \
+  -d @debezium-postgres.json
+```
+
+---
+
+## 📦 Transactional Outbox Pattern Implementation
+
+### Step 1: Outbox Event Table Schema
+
+```sql
+CREATE TABLE outbox_events (
+    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+    aggregate_type VARCHAR(64) NOT NULL,
+    aggregate_id VARCHAR(64) NOT NULL,
+    event_type VARCHAR(64) NOT NULL,
+    payload JSONB NOT NULL,
+    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
+);
+
+-- Index for optional periodic cleanup
+CREATE INDEX idx_outbox_created_at ON outbox_events (created_at);
+```
+
+### Step 2: Atomic Business Transaction (TypeScript / Node.js)
+
+```typescript
+import { PoolClient } from 'pg';
+
+interface OrderEvent {
+  orderId: string;
+  userId: string;
+  totalCents: number;
+}
+
+export async function createOrderWithOutbox(client: PoolClient, event: OrderEvent): Promise<void> {
+  try {
+    await client.query('BEGIN');
+
+    // 1. Mutate primary business state
+    const orderSql = `
+      INSERT INTO orders (id, user_id, total_cents, status)
+      VALUES ($1, $2, $3, 'CREATED')
+    `;
+    await client.query(orderSql, [event.orderId, event.userId, event.totalCents]);
+
+    // 2. Append event to outbox within same transaction
+    const outboxSql = `
+      INSERT INTO outbox_events (aggregate_type, aggregate_id, event_type, payload)
+      VALUES ('Order', $1, 'OrderCreated', $2)
+    `;
+    await client.query(outboxSql, [event.orderId, JSON.stringify(event)]);
+
+    await client.query('COMMIT');
+  } catch (error) {
+    await client.query('ROLLBACK');
+    throw error;
+  }
+}
+```
+
+---
+
+## 🔄 Debezium Event Payload Structure
+
+```json
+{
+  "before": null,
+  "after": {
+    "id": "f47ac10b-58cc-4372-a567-0e02b2c3d479",
+    "user_id": "usr_98231",
+    "total_cents": 12500,
+    "status": "CREATED"
+  },
+  "source": {
+    "version": "2.5.0.Final",
+    "connector": "postgresql",
+    "name": "prod_cdc",
+    "ts_ms": 1718000000000,
+    "snapshot": "false",
+    "db": "production_app",
+    "schema": "public",
+    "table": "orders",
+    "txId": 984120,
+    "lsn": 104857600
+  },
+  "op": "c",
+  "ts_ms": 1718000000100
+}
+```
+
+* `op`: `"c"` (create), `"u"` (update), `"d"` (delete), `"r"` (snapshot read).
+
+---
+
+## 🛡️ Production Failure Modes & Operational Checklist
+
+1. **Replication Slot Disk Exhaustion**:
+   - *Risk*: If Kafka Connect crashes, Postgres retains WAL logs indefinitely, filling disk.
+   - *Mitigation*: Set `max_slot_wal_keep_size = 50GB` in `postgresql.conf` to protect database uptime.
+2. **Heartbeat Tuning**:
+   - Always configure `heartbeat.interval.ms` (e.g. 5000ms) to ensure LSN progression even during low-traffic periods.
+3. **Schema Evolution Handling**:
+   - Use Schema Registry (Avro or Protobuf) with `BACKWARD` or `FULL` compatibility to avoid downstream deserialization failures.

package/skills/docker-compose-orchestrator/SKILL.md

@@ -0,0 +1,85 @@
+---
+name: docker-compose-orchestrator
+description: Architects, configures, and troubleshoots multi-container Docker Compose environments, database services (Postgres, pgvector, Redis), healthchecks, and local dev networking.
+---
+
+# Docker Compose Multi-Service Orchestrator
+
+This skill guides the design, hardening, and troubleshooting of multi-container local and staging environments using Docker Compose v2.
+
+---
+
+## 1. Core Principles
+
+1. **Deterministic Healthchecks**:
+   - Always define explicit `healthcheck` blocks for stateful services (databases, caches, message brokers).
+   - Use `depends_on: { condition: service_healthy }` to prevent app containers from starting before databases are ready.
+2. **Persistent Named Volumes**:
+   - Store persistent data (e.g., PostgreSQL data, Redis dumps, uploaded media) in named volumes, never in ephemeral container layers.
+3. **Environment Isolation**:
+   - Use `.env.example` templates with sensible local defaults; never hardcode production secrets in compose files.
+
+---
+
+## 2. Production-Grade Compose Blueprint
+
+Example architecture pairing an application server with PostgreSQL (pgvector enabled) and Redis caching:
+
+```yaml
+version: '3.8'
+
+services:
+  db:
+    image: pgvector/pgvector:pg16
+    container_name: app_db
+    environment:
+      POSTGRES_USER: ${POSTGRES_USER:-postgres}
+      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-postgres}
+      POSTGRES_DB: ${POSTGRES_DB:-app_dev}
+    ports:
+      - "5432:5432"
+    volumes:
+      - pgdata:/var/lib/postgresql/data
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U postgres"]
+      interval: 5s
+      timeout: 5s
+      retries: 5
+    networks:
+      - app-net
+
+  redis:
+    image: redis:7-alpine
+    container_name: app_redis
+    ports:
+      - "6379:6379"
+    volumes:
+      - redisdata:/data
+    healthcheck:
+      test: ["CMD", "redis-cli", "ping"]
+      interval: 5s
+      timeout: 3s
+      retries: 5
+    networks:
+      - app-net
+
+volumes:
+  pgdata:
+  redisdata:
+
+networks:
+  app-net:
+    driver: bridge
+```
+
+---
+
+## 3. Common Operations & Diagnostics
+
+| Command | Action |
+| :--- | :--- |
+| `docker compose up -d` | Start all services in the background |
+| `docker compose ps` | Inspect service health and exposed ports |
+| `docker compose logs -f <service>` | Stream logs for a specific container |
+| `docker compose down -v` | Tear down containers and wipe persistent volumes |
+| `docker compose exec <service> sh` | Open an interactive shell inside a container |

package/skills/duckdb-local-olap/SKILL.md

@@ -0,0 +1,134 @@
+---
+name: duckdb-local-olap
+description: >-
+  Architects, queries, and optimizes embedded in-process DuckDB analytical workflows, direct Parquet/S3 querying, zero-copy Apache Arrow dataframes, and MotherDuck hybrid cloud execution.
+---
+
+# DuckDB Local OLAP & Serverless Analytics
+
+This skill guides building high-speed analytical engines using DuckDB—the "SQLite for analytics"—enabling columnar vectorized execution, direct querying of compressed Parquet files on cloud storage, and zero-copy transfers with Apache Arrow.
+
+---
+
+## ⚡ Core Patterns
+
+### 1. Direct Parquet Querying Over S3 Without Ingestion
+
+Query gigabytes of remote compressed files with projection and predicate pushdowns (only requested byte ranges are transferred over the network):
+
+```sql
+-- 1. Install and load HTTP and S3 extensions
+INSTALL httpfs;
+LOAD httpfs;
+
+-- 2. Configure S3 authentication
+SET s3_region = 'us-east-1';
+SET s3_access_key_id = 'YOUR_ACCESS_KEY';
+SET s3_secret_access_key = 'YOUR_SECRET_KEY';
+
+-- 3. Query remote Parquet directory with globs
+SELECT
+    country,
+    count(*) AS total_transactions,
+    round(avg(amount_usd), 2) AS avg_amount,
+    quantile_cont(amount_usd, 0.95) AS p95_amount
+FROM read_parquet('s3://my-data-lake/transactions/year=2026/month=*/*.parquet')
+WHERE status = 'settled'
+GROUP BY country
+ORDER BY total_transactions DESC
+LIMIT 20;
+```
+
+---
+
+### 2. Zero-Copy Python & Polars / Arrow Integration
+
+Pass memory buffers directly between DuckDB's vectorized engine and Polars or PyArrow without JSON or IPC serialization:
+
+```python
+import duckdb
+import polars as pl
+
+# Connect to in-memory instance
+con = duckdb.connect()
+
+# Set aggressive memory limits and worker threads
+con.execute("SET memory_limit = '8GB';")
+con.execute("SET threads = 8;")
+
+# Create a sample Polars DataFrame
+df = pl.DataFrame({
+    "user_id": [101, 102, 103],
+    "scores": [98.5, 76.2, 89.0]
+})
+
+# Query Polars DataFrame directly as a virtual table (zero-copy)
+arrow_table = con.execute("""
+    SELECT 
+        user_id,
+        scores,
+        CASE WHEN scores >= 90 THEN 'A' ELSE 'B' END AS grade
+    FROM df
+    WHERE scores > 80
+""").arrow()
+
+# Convert back to Polars instantaneously
+result_df = pl.from_arrow(arrow_table)
+print(result_df)
+```
+
+---
+
+### 3. Node.js Embedded Analytics Engine
+
+```typescript
+import { Database } from 'duckdb-async';
+
+export async function aggregateLocalParquet(parquetPath: string) {
+  const db = await Database.create(':memory:');
+  
+  const rows = await db.all(`
+    SELECT 
+        category,
+        count(*) AS item_count,
+        sum(price) AS gross_revenue
+    FROM read_parquet(?)
+    GROUP BY category
+    ORDER BY gross_revenue DESC
+  `, [parquetPath]);
+
+  await db.close();
+  return rows;
+}
+```
+
+---
+
+### 4. MotherDuck Cloud Hybrid Execution
+
+Scale seamlessly from a single laptop to collaborative cloud storage without modifying SQL queries:
+
+```python
+import duckdb
+
+# Connect to MotherDuck (cloud database)
+con = duckdb.connect("md:production_analytics")
+
+# Join local disk Parquet files directly with MotherDuck cloud tables
+con.execute("""
+    SELECT 
+        cloud_users.organization_id,
+        count(local_events.event_id) AS event_volume
+    FROM read_parquet('./local_cache/*.parquet') AS local_events
+    JOIN cloud_users ON local_events.user_id = cloud_users.id
+    GROUP BY cloud_users.organization_id
+""")
+```
+
+---
+
+## 🛠️ Performance Checklist
+
+- **Use Parquet Over CSV**: Parquet is 10x-50x faster due to columnar compression, statistics dictionaries, and column skipping.
+- **Configure Memory Limit**: Always set `SET memory_limit = 'XGB';` to ensure DuckDB uses external sorting/joining when data exceeds RAM instead of crashing the process.
+- **Avoid Row-by-Row Inserts**: Use `Appender` or `COPY ... FROM ...` in bulk batches of 50,000+ rows.