everything-gemini 1.2.0 → 1.9.0

package/skills/cloudflare-r2-object-storage/SKILL.md

@@ -0,0 +1,143 @@
+---
+name: cloudflare-r2-object-storage
+description: >-
+  Architects, configures, and integrates Cloudflare R2 zero-egress object storage using S3-compatible APIs, pre-signed upload URLs, multipart streaming, and edge CDN distribution.
+---
+
+# Cloudflare R2 Object Storage & Edge Distribution
+
+This skill guides the implementation of zero-egress fee object storage using Cloudflare R2 with S3-compatible APIs, pre-signed URLs, multipart uploads, and edge CDN distribution.
+
+---
+
+## ⚡ Architectural Comparison: R2 vs Standard S3
+
+| Feature | Cloudflare R2 | Standard AWS S3 |
+| :--- | :--- | :--- |
+| **Data Egress Fees** | **$0.00 / GB (Free)** | ~$0.09 / GB |
+| **API Compatibility** | Full S3 API v4 signature | Native AWS S3 |
+| **Edge Integration** | Native Cloudflare Workers & CDN | CloudFront integration required |
+| **Storage Pricing** | $0.015 / GB-month | $0.023 / GB-month |
+| **Global Replication** | Automatic across edge locations | Cross-Region Replication (CRR) billed |
+
+---
+
+## 🛠️ TypeScript S3 Client Configuration (`@aws-sdk/client-s3`)
+
+R2 endpoints use account-specific URLs with AWS Signature Version 4:
+
+```typescript
+import { S3Client, PutObjectCommand, GetObjectCommand } from '@aws-sdk/client-s3';
+import { getSignedUrl } from '@aws-sdk/s3-request-presigner';
+
+const ACCOUNT_ID = process.env.CLOUDFLARE_ACCOUNT_ID!;
+const R2_ACCESS_KEY_ID = process.env.R2_ACCESS_KEY_ID!;
+const R2_SECRET_ACCESS_KEY = process.env.R2_SECRET_ACCESS_KEY!;
+
+export const r2Client = new S3Client({
+  region: 'auto',
+  endpoint: `https://${ACCOUNT_ID}.r2.cloudflarestorage.com`,
+  credentials: {
+    accessKeyId: R2_ACCESS_KEY_ID,
+    secretAccessKey: R2_SECRET_ACCESS_KEY,
+  },
+});
+```
+
+---
+
+## 📤 Pre-Signed URLs for Direct Browser Uploads
+
+Prevent API server bottlenecking by generating short-lived pre-signed upload URLs:
+
+```typescript
+export async function generateUploadUrl(key: string, contentType: string, expiresInSeconds = 300) {
+  const command = new PutObjectCommand({
+    Bucket: 'enterprise-artifacts',
+    Key: key,
+    ContentType: contentType,
+  });
+
+  const uploadUrl = await getSignedUrl(r2Client, command, { expiresIn: expiresInSeconds });
+  return {
+    uploadUrl,
+    key,
+    publicUrl: `https://cdn.example.com/${key}`,
+  };
+}
+```
+
+---
+
+## 📦 High-Throughput Multipart Uploads (Node.js / Python)
+
+For files exceeding 100MB, use parallel multipart uploads with the Upload helper:
+
+```typescript
+import { Upload } from '@aws-sdk/lib-storage';
+import fs from 'node:fs';
+
+export async function uploadLargeArchive(filePath: string, destinationKey: string) {
+  const fileStream = fs.createReadStream(filePath);
+  
+  const parallelUpload = new Upload({
+    client: r2Client,
+    params: {
+      Bucket: 'enterprise-artifacts',
+      Key: destinationKey,
+      Body: fileStream,
+      ContentType: 'application/octet-stream',
+    },
+    queueSize: 4,          // 4 concurrent part uploads
+    partSize: 10 * 1024 * 1024, // 10MB chunk size
+    leavePartsOnError: false,
+  });
+
+  parallelUpload.on('httpUploadProgress', (progress) => {
+    console.log(`Uploaded ${progress.loaded} of ${progress.total} bytes`);
+  });
+
+  return await parallelUpload.done();
+}
+```
+
+---
+
+## 🌐 Custom Domain & Cloudflare CDN Caching Rule
+
+Expose R2 buckets through a custom domain with edge cache headers:
+
+```javascript
+// Cloudflare Worker Edge Proxy
+export default {
+  async fetch(request, env) {
+    const url = new URL(request.url);
+    const key = url.pathname.slice(1);
+
+    if (request.method !== 'GET') {
+      return new Response('Method Not Allowed', { status: 405 });
+    }
+
+    const object = await env.MY_R2_BUCKET.get(key);
+    if (!object) {
+      return new Response('Not Found', { status: 404 });
+    }
+
+    const headers = new Headers();
+    object.writeHttpMetadata(headers);
+    headers.set('etag', object.httpEtag);
+    headers.set('Cache-Control', 'public, max-age=31536000, immutable');
+
+    return new Response(object.body, { headers });
+  },
+};
+```
+
+---
+
+## 📋 Architectural Rules & Best Practices
+
+1. **Bucket Region**: Always set region to `auto` in AWS S3 SDK clients; R2 automatically routes storage to the lowest-latency regional cluster.
+2. **CORS Headers**: Set permissive CORS rules (`AllowedOrigins`, `AllowedMethods: ["PUT", "GET"]`) on the R2 bucket for direct browser pre-signed uploads.
+3. **Multipart Cleanup**: Configure bucket lifecycle rules to automatically abort incomplete multipart uploads after 7 days (`AbortIncompleteMultipartUpload`).
+4. **Zero-Egress Strategy**: Route public media and file downloads through Cloudflare Custom Domains or Workers to benefit from zero bandwidth charges.

package/skills/dbt-transformation/SKILL.md

@@ -0,0 +1,179 @@
+---
+name: dbt-transformation
+description: >-
+  Designs, develops, and tests modular SQL data transformations using dbt (data build tool), Jinja macros, semantic layer metrics, automated schema tests, and documentation generation.
+---
+
+# dbt Modular Transformation & Data Modeling
+
+This skill guides structuring production-grade ELT pipelines using dbt (Core & Cloud), enforcing modular layer architecture (staging, intermediate, marts), DRY Jinja macros, automated data testing, and incremental table materialization.
+
+---
+
+## ⚡ Core Patterns
+
+### 1. Multi-Layer Dimensional Modeling Architecture
+
+Organize transformations into clean, progressive DAG layers:
+
+```text
+models/
+├── staging/                     # 1:1 view on raw sources (renaming, casting)
+│   ├── stripe/
+│   │   ├── _stripe__sources.yml
+│   │   ├── _stripe__models.yml
+│   │   └── stg_stripe__payments.sql
+├── intermediate/                # Business logic joins & window aggregations
+│   └── finance/
+│       ├── _int_finance__models.yml
+│       └── int_payments_pivoted_to_orders.sql
+└── marts/                       # Dimensional star schema (facts & dimensions)
+    ├── core/
+    │   ├── fct_orders.sql
+    │   └── dim_customers.sql
+    └── marketing/
+        └── fct_campaign_conversions.sql
+```
+
+---
+
+### 2. Staging Layer Model with CTE Standards
+
+Clean raw source tables into normalized relational types:
+
+```sql
+-- models/staging/stripe/stg_stripe__payments.sql
+with source as (
+    select * from {{ source('stripe', 'payment') }}
+),
+
+renamed as (
+    select
+        id as payment_id,
+        orderid as order_id,
+        paymentmethod as payment_method,
+        status as payment_status,
+        -- Amount stored in cents; convert to dollars
+        round(amount / 100.0, 2) as amount_usd,
+        created as created_at
+    from source
+)
+
+select * from renamed
+```
+
+Declare tests and column descriptions in accompanying YAML:
+
+```yaml
+# models/staging/stripe/_stripe__models.yml
+version: 2
+
+models:
+  - name: stg_stripe__payments
+    description: "Normalized payments table from Stripe webhook ingest."
+    columns:
+      - name: payment_id
+        description: "Primary key for payment events."
+        tests:
+          - unique
+          - not_null
+      - name: payment_status
+        tests:
+          - accepted_values:
+              values: ['success', 'fail', 'pending']
+```
+
+---
+
+### 3. High-Performance Incremental Fact Table
+
+Process only newly ingested or updated records using `is_incremental()`:
+
+```sql
+-- models/marts/core/fct_orders.sql
+{{
+    config(
+        materialized='incremental',
+        unique_key='order_id',
+        incremental_strategy='merge',
+        on_schema_change='fail'
+    )
+}}
+
+with orders as (
+    select * from {{ ref('stg_jaffle_shop__orders') }}
+    {% if is_incremental() %}
+        -- Only query rows created or modified in the last 3 days for backfill resilience
+        where order_date >= (select dateadd('day', -3, max(order_date)) from {{ this }})
+    {% endif %}
+),
+
+payments as (
+    select * from {{ ref('stg_stripe__payments') }}
+),
+
+order_payments as (
+    select
+        order_id,
+        sum(case when payment_status = 'success' then amount_usd else 0 end) as amount_usd
+    from payments
+    group by 1
+),
+
+final as (
+    select
+        orders.order_id,
+        orders.customer_id,
+        orders.order_date,
+        orders.status as order_status,
+        coalesce(order_payments.amount_usd, 0) as total_amount_usd
+    from orders
+    left join order_payments on orders.order_id = order_payments.order_id
+)
+
+select * from final
+```
+
+---
+
+### 4. Custom Jinja Macro for Dynamic Pivoting
+
+Avoid repetitive boilerplate using parameterized Jinja macros:
+
+```sql
+-- macros/pivot_sum.sql
+{% macro pivot_sum(column_name, values_list, aggregate_col='amount_usd') %}
+    {% for val in values_list %}
+        sum(case when {{ column_name }} = '{{ val }}' then {{ aggregate_col }} else 0 end) as {{ val }}_amount
+        {% if not loop.last %},{% endif %}
+    {% endfor %}
+{% endmacro %}
+```
+
+Usage in intermediate model:
+
+```sql
+select
+    order_id,
+    {{ pivot_sum('payment_method', ['credit_card', 'coupon', 'bank_transfer', 'gift_card']) }}
+from {{ ref('stg_stripe__payments') }}
+group by 1
+```
+
+---
+
+## 🎯 Verification & Diagnostics
+
+1. **Compile SQL with Resolved Jinja**:
+   ```bash
+   dbt compile --select fct_orders
+   ```
+2. **Execute Targeted Subgraph**:
+   ```bash
+   dbt run --select +fct_orders       # Run fct_orders and all upstream dependencies
+   dbt test --select stg_stripe+      # Test staging models and downstream marts
+   ```
+3. **Build Documentation & DAG Visualizer**:
+   ```bash
+   dbt docs generate && dbt docs serve
+   ```

package/skills/debezium-cdc/SKILL.md

@@ -0,0 +1,166 @@
+---
+name: debezium-cdc
+description: >-
+  Change Data Capture (CDC) streaming architectures using Debezium, PostgreSQL logical decoding (pgoutput), MySQL binlog replication, Kafka Connect, and transactional outbox patterns. Use when capturing real-time database mutations, synchronizing search indices, or avoiding dual-write race conditions.
+---
+
+# Debezium Change Data Capture (CDC) Architecture
+
+A production runbook for implementing distributed **Change Data Capture (CDC)** pipelines using Debezium, PostgreSQL/MySQL WAL replication, Kafka Connect, and event-driven data streaming.
+
+---
+
+## 🎯 Core Architectural Principles
+
+1. **Log-Based CDC Over Polling**: Reads write-ahead logs (WAL / binlog) directly at the storage engine level, incurring minimal CPU overhead and zero query locks on primary tables.
+2. **Transactional Outbox Pattern**: Guarantees atomic database operations and event emission within a single ACID transaction without two-phase commit (2PC) or dual-write inconsistencies.
+3. **Strict Ordering Guarantees**: Partitioning by primary key ensures sequential ordering per record entity across consumer groups.
+4. **Idempotent Consumers**: Design downstream event processors with upsert semantics and revision monotonic version checks.
+
+---
+
+## 🏗️ Debezium PostgreSQL Connector Configuration
+
+### 1. Database Prerequisite (`postgresql.conf`)
+
+```ini
+# Enable logical replication
+wal_level = logical
+max_wal_senders = 10
+max_replication_slots = 10
+```
+
+### 2. Kafka Connect Connector Payload (`debezium-postgres.json`)
+
+```json
+{
+  "name": "postgres-order-cdc",
+  "config": {
+    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
+    "tasks.max": "1",
+    "plugin.name": "pgoutput",
+    "database.hostname": "postgres.internal",
+    "database.port": "5432",
+    "database.user": "debezium_cdc",
+    "database.password": "${file:/secrets/db-credentials.properties:password}",
+    "database.dbname": "production_app",
+    "database.server.name": "prod_cdc",
+    "table.include.list": "public.orders,public.outbox_events",
+    "slot.name": "debezium_orders_slot",
+    "publication.name": "dbz_orders_pub",
+    "publication.autocreate.mode": "filtered",
+    "decimal.handling.mode": "double",
+    "tombstones.on.delete": "true",
+    "heartbeat.interval.ms": "5000"
+  }
+}
+```
+
+Deploy to Kafka Connect cluster via REST API:
+
+```bash
+curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" \
+  http://kafka-connect.internal:8083/connectors/ \
+  -d @debezium-postgres.json
+```
+
+---
+
+## 📦 Transactional Outbox Pattern Implementation
+
+### Step 1: Outbox Event Table Schema
+
+```sql
+CREATE TABLE outbox_events (
+    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+    aggregate_type VARCHAR(64) NOT NULL,
+    aggregate_id VARCHAR(64) NOT NULL,
+    event_type VARCHAR(64) NOT NULL,
+    payload JSONB NOT NULL,
+    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
+);
+
+-- Index for optional periodic cleanup
+CREATE INDEX idx_outbox_created_at ON outbox_events (created_at);
+```
+
+### Step 2: Atomic Business Transaction (TypeScript / Node.js)
+
+```typescript
+import { PoolClient } from 'pg';
+
+interface OrderEvent {
+  orderId: string;
+  userId: string;
+  totalCents: number;
+}
+
+export async function createOrderWithOutbox(client: PoolClient, event: OrderEvent): Promise<void> {
+  try {
+    await client.query('BEGIN');
+
+    // 1. Mutate primary business state
+    const orderSql = `
+      INSERT INTO orders (id, user_id, total_cents, status)
+      VALUES ($1, $2, $3, 'CREATED')
+    `;
+    await client.query(orderSql, [event.orderId, event.userId, event.totalCents]);
+
+    // 2. Append event to outbox within same transaction
+    const outboxSql = `
+      INSERT INTO outbox_events (aggregate_type, aggregate_id, event_type, payload)
+      VALUES ('Order', $1, 'OrderCreated', $2)
+    `;
+    await client.query(outboxSql, [event.orderId, JSON.stringify(event)]);
+
+    await client.query('COMMIT');
+  } catch (error) {
+    await client.query('ROLLBACK');
+    throw error;
+  }
+}
+```
+
+---
+
+## 🔄 Debezium Event Payload Structure
+
+```json
+{
+  "before": null,
+  "after": {
+    "id": "f47ac10b-58cc-4372-a567-0e02b2c3d479",
+    "user_id": "usr_98231",
+    "total_cents": 12500,
+    "status": "CREATED"
+  },
+  "source": {
+    "version": "2.5.0.Final",
+    "connector": "postgresql",
+    "name": "prod_cdc",
+    "ts_ms": 1718000000000,
+    "snapshot": "false",
+    "db": "production_app",
+    "schema": "public",
+    "table": "orders",
+    "txId": 984120,
+    "lsn": 104857600
+  },
+  "op": "c",
+  "ts_ms": 1718000000100
+}
+```
+
+* `op`: `"c"` (create), `"u"` (update), `"d"` (delete), `"r"` (snapshot read).
+
+---
+
+## 🛡️ Production Failure Modes & Operational Checklist
+
+1. **Replication Slot Disk Exhaustion**:
+   - *Risk*: If Kafka Connect crashes, Postgres retains WAL logs indefinitely, filling disk.
+   - *Mitigation*: Set `max_slot_wal_keep_size = 50GB` in `postgresql.conf` to protect database uptime.
+2. **Heartbeat Tuning**:
+   - Always configure `heartbeat.interval.ms` (e.g. 5000ms) to ensure LSN progression even during low-traffic periods.
+3. **Schema Evolution Handling**:
+   - Use Schema Registry (Avro or Protobuf) with `BACKWARD` or `FULL` compatibility to avoid downstream deserialization failures.

package/skills/distributed-rate-limiting/SKILL.md

@@ -0,0 +1,138 @@
+---
+name: distributed-rate-limiting
+description: >-
+  Architects, implements, and tunes distributed rate limiting using Redis sliding window logs, token buckets, tiered API quotas, RFC 6585 headers, and circuit breakers.
+---
+
+# Distributed Rate Limiting & API Traffic Shaping
+
+This skill guides the design, implementation, and tuning of distributed rate limiting systems across microservice clusters using Redis, token buckets, and sliding window algorithms.
+
+---
+
+## ⚡ Algorithm Comparison Matrix
+
+| Algorithm | Precision | Memory Usage | Burst Handling | Best For |
+| :--- | :--- | :--- | :--- | :--- |
+| **`Sliding Window Log`** | 100% Exact | High ($O(N)$ requests) | Excellent | Low-to-medium traffic sensitive endpoints (auth, payments). |
+| **`Sliding Window Counter`**| ~99% Exact | Fixed ($O(1)$) | Smooth | High-scale public APIs and ingress gateways. |
+| **`Token Bucket`** | High | Very Low ($O(1)$) | Controlled bursts | Standard microservice backpressuring and rate smoothing. |
+| **`Fixed Window`** | Poor (2x burst at boundary) | Lowest | Prone to spikes | Simple non-critical background jobs only. |
+
+---
+
+## 🔒 Atomic Sliding Window Rate Limiter (Redis Lua)
+
+Atomic execution in Redis ensures no race conditions across distributed API instances:
+
+```lua
+-- sliding_window.lua
+-- KEYS[1]: Rate limit key (e.g., "ratelimit:tenant_123:orders")
+-- ARGV[1]: Current UNIX timestamp in milliseconds
+-- ARGV[2]: Window size in milliseconds (e.g., 60000 for 1 minute)
+-- ARGV[3]: Maximum allowed requests in window (e.g., 100)
+
+local key = KEYS[1]
+local now = tonumber(ARGV[1])
+local window = tonumber(ARGV[2])
+local limit = tonumber(ARGV[3])
+local clearBefore = now - window
+
+-- 1. Remove expired request timestamps outside the window
+redis.call('ZREMRANGEBYSCORE', key, 0, clearBefore)
+
+-- 2. Count requests currently in the sliding window
+local currentRequests = redis.call('ZCARD', key)
+
+if currentRequests < limit then
+    -- 3. Register current request with unique timestamp as score and member
+    redis.call('ZADD', key, now, now .. '-' .. math.random(1000, 9999))
+    redis.call('PEXPIRE', key, window)
+    return { 1, limit - currentRequests - 1, 0 } -- [Allowed: 1, Remaining, RetryAfter: 0]
+else
+    -- 4. Calculate time until oldest entry expires
+    local oldest = redis.call('ZRANGE', key, 0, 0, 'WITHSCORES')
+    local retryAfter = math.ceil((tonumber(oldest[2]) + window - now) / 1000)
+    return { 0, 0, math.max(1, retryAfter) }      -- [Allowed: 0, Remaining: 0, RetryAfter]
+end
+```
+
+---
+
+## 🌐 Express / Fastify / Node.js Middleware Implementation
+
+Wrap Redis Lua execution into production middleware with RFC 6585 compliance:
+
+```typescript
+import { Request, Response, NextFunction } from 'express';
+import Redis from 'ioredis';
+
+const redis = new Redis(process.env.REDIS_URL || 'redis://localhost:6379');
+
+export function createRateLimiter(options: { windowMs: number; max: number; keyGenerator?: (req: Request) => string }) {
+  return async (req: Request, res: Response, next: NextFunction) => {
+    const key = options.keyGenerator ? options.keyGenerator(req) : `ratelimit:${req.ip}`;
+    const now = Date.now();
+
+    try {
+      const [allowed, remaining, retryAfter] = await redis.eval(
+        SLIDING_WINDOW_LUA_SCRIPT,
+        1,
+        key,
+        now.toString(),
+        options.windowMs.toString(),
+        options.max.toString()
+      ) as [number, number, number];
+
+      // Set standard RFC 6585 RateLimit response headers
+      res.setHeader('X-RateLimit-Limit', options.max);
+      res.setHeader('X-RateLimit-Remaining', remaining);
+      res.setHeader('X-RateLimit-Reset', Math.ceil((now + options.windowMs) / 1000));
+
+      if (allowed === 1) {
+        return next();
+      }
+
+      res.setHeader('Retry-After', retryAfter);
+      return res.status(429).json({
+        error: 'Too Many Requests',
+        message: `Rate limit exceeded. Please retry after ${retryAfter} seconds.`,
+        retryAfter,
+      });
+    } catch (err) {
+      // Circuit Breaker: Fail-Open for high availability if Redis degrades
+      console.error('RateLimiter Redis failure, failing open:', err);
+      return next();
+    }
+  };
+}
+```
+
+---
+
+## 🎯 Tiered Multi-Dimensional Quotas
+
+Configure tiered rate limits based on authenticated API keys and endpoint criticality:
+
+```typescript
+export function getTierLimits(tier: 'free' | 'pro' | 'enterprise') {
+  switch (tier) {
+    case 'enterprise':
+      return { windowMs: 60_000, max: 10_000 };
+    case 'pro':
+      return { windowMs: 60_000, max: 1_200 };
+    case 'free':
+    default:
+      return { windowMs: 60_000, max: 60 };
+  }
+}
+```
+
+---
+
+## 📋 Architectural Rules & Best Practices
+
+1. **Atomic Scripts**: Never implement rate limiters with separated `GET` and `INCR` commands; always execute atomic Lua scripts or Redis Transactions (`MULTI/EXEC`) to prevent race conditions under load.
+2. **Fail-Open Strategy**: For user-facing critical flows, catch Redis connection timeouts and fail-open with alerting rather than causing total service outages.
+3. **Multi-Dimensional Keys**: Combine client identity with endpoint routes (e.g. `tenant:123:POST:/api/v1/checkout`) to prevent denial of service on high-cost endpoints.
+4. **Header Standards**: Always emit `Retry-After`, `X-RateLimit-Limit`, and `X-RateLimit-Remaining` to allow compliant clients to back off exponentially.