everything-gemini 1.2.0 → 1.8.0

package/skills/axum-web-service/SKILL.md

@@ -0,0 +1,159 @@
+---
+name: axum-web-service
+description: >-
+  Architects, builds, and optimizes production Rust web APIs using Axum, Tokio async runtime, Tower middlewares, type-safe state extractors, and SQLx connection pools.
+---
+
+# Axum Web Service Architecture (Rust)
+
+This skill guides engineering microservices with Axum—the high-throughput, ergonomic Rust web framework built on top of Tokio, Tower, and Hyper.
+
+---
+
+## ⚡ Core Patterns
+
+### 1. Application State & Router Composition (`main.rs`)
+
+```rust
+use axum::{
+    extract::{Path, State},
+    http::StatusCode,
+    response::IntoResponse,
+    routing::{get, post},
+    Json, Router,
+};
+use sqlx::PgPool;
+use std::{net::SocketAddr, sync::Arc};
+use tower_http::{cors::CorsLayer, trace::TraceLayer};
+
+#[derive(Clone)]
+pub struct AppState {
+    pub db: PgPool,
+    pub redis: redis::aio::ConnectionManager,
+}
+
+#[tokio::main]
+async fn main() -> Result<(), Box<dyn std::error::Error>> {
+    tracing_subscriber::fmt::init();
+
+    let db_pool = PgPool::connect(&std::env::var("DATABASE_URL")?).await?;
+    let redis_client = redis::Client::open(std::env::var("REDIS_URL")?)?;
+    let redis_mgr = redis_client.get_tokio_connection_manager().await?;
+
+    let shared_state = Arc::new(AppState {
+        db: db_pool,
+        redis: redis_mgr,
+    });
+
+    let app = Router::new()
+        .route("/healthz", get(|| async { StatusCode::OK }))
+        .nest("/api/v1", api_v1_router())
+        .layer(TraceLayer::new_for_http())
+        .layer(CorsLayer::permissive())
+        .with_state(shared_state);
+
+    let addr = SocketAddr::from(([0, 0, 0, 0], 8080));
+    let listener = tokio::net::TcpListener::bind(addr).await?;
+    tracing::info!("Listening on {}", addr);
+    axum::serve(listener, app).await?;
+
+    Ok(())
+}
+```
+
+---
+
+### 2. Type-Safe Handlers with SQLx (`handlers/users.rs`)
+
+Use Axum's compile-time extractors to ensure payload and state invariants before handler execution:
+
+```rust
+use axum::{
+    extract::{Path, State},
+    Json,
+};
+use serde::{Deserialize, Serialize};
+use std::sync::Arc;
+use crate::{AppState, error::AppError};
+
+#[derive(Deserialize)]
+pub struct CreateUserPayload {
+    pub username: String,
+    pub email: String,
+}
+
+#[derive(Serialize)]
+pub struct UserResponse {
+    pub id: i64,
+    pub username: String,
+    pub email: String,
+}
+
+pub async fn create_user(
+    State(state): State<Arc<AppState>>,
+    Json(payload): Json<CreateUserPayload>,
+) -> Result<Json<UserResponse>, AppError> {
+    // Compile-time verified query via SQLx
+    let user = sqlx::query_as!(
+        UserResponse,
+        r#"
+        INSERT INTO users (username, email)
+        VALUES ($1, $2)
+        RETURNING id, username, email
+        "#,
+        payload.username,
+        payload.email
+    )
+    .fetch_one(&state.db)
+    .await?;
+
+    Ok(Json(user))
+}
+```
+
+---
+
+### 3. Unified RFC 7807 Error Envelope (`error.rs`)
+
+Transform any internal database or validation failure into clean, typed JSON responses using `IntoResponse`:
+
+```rust
+use axum::{
+    http::StatusCode,
+    response::{IntoResponse, Response},
+    Json,
+};
+use serde_json::json;
+
+#[derive(thiserror::Error, Debug)]
+pub enum AppError {
+    #[error("Database error: {0}")]
+    Database(#[from] sqlx::Error),
+    #[error("Resource not found")]
+    NotFound,
+    #[error("Validation failed: {0}")]
+    Validation(String),
+}
+
+impl IntoResponse for AppError {
+    fn into_response(self) -> Response {
+        let (status, detail) = match self {
+            AppError::NotFound => (StatusCode::NOT_FOUND, "The requested resource does not exist.".to_string()),
+            AppError::Validation(msg) => (StatusCode::BAD_REQUEST, msg),
+            AppError::Database(err) => {
+                tracing::error!("Internal DB error: {:?}", err);
+                (StatusCode::INTERNAL_SERVER_ERROR, "An internal server error occurred.".to_string())
+            }
+        };
+
+        let body = Json(json!({
+            "type": "https://errors.example.com",
+            "title": status.canonical_reason().unwrap_or("Error"),
+            "status": status.as_u16(),
+            "detail": detail
+        }));
+
+        (status, body).into_response()
+    }
+}
+```

package/skills/cilium-ebpf-networking/SKILL.md

@@ -0,0 +1,145 @@
+---
+name: cilium-ebpf-networking
+description: >-
+  Cloud-native eBPF networking, Kubernetes CNI replacement, transparent mutual TLS (mTLS), L7 CiliumNetworkPolicies, and Hubble network observability. Use when securing cluster traffic, eliminating kube-proxy iptables bottlenecks, or debugging service mesh latency.
+---
+
+# Cilium eBPF Networking & Security Architecture
+
+A production runbook for deploying, configuring, and troubleshooting **Cilium eBPF**, high-performance Kubernetes networking, and deep packet observability with **Hubble**.
+
+---
+
+## 🎯 Architectural Advantages of eBPF
+
+1. **Bypasses iptables / IPVS Overhead**: Programs run directly in the Linux kernel network stack (`tc` / `xdp`), providing O(1) connection routing regardless of service count.
+2. **Kernel-Level Identity**: Enforces security policies based on cryptographically verified Kubernetes metadata and SPIFFE identities rather than volatile IP addresses.
+3. **Transparent Encryption**: Mutual TLS (mTLS) or WireGuard encryption negotiated dynamically at the kernel layer with zero application sidecars.
+4. **Hubble Observability**: Zero-overhead flow logging, DNS resolution tracking, and L7 HTTP/gRPC metrics without code instrumentation.
+
+---
+
+## 🚀 Helm Deployment (kube-proxy Replacement)
+
+```bash
+helm repo add cilium https://helm.cilium.io/
+helm repo update
+
+helm install cilium cilium/cilium --version 1.16.0 \
+  --namespace kube-system \
+  --set kubeProxyReplacement=true \
+  --set k8sServiceHost=10.0.0.1 \
+  --set k8sServicePort=6443 \
+  --set routingMode=native \
+  --set autoDirectNodeRoutes=true \
+  --set bpf.masquerade=true \
+  --set encryption.enabled=true \
+  --set encryption.type=wireguard \
+  --set hubble.enabled=true \
+  --set hubble.relay.enabled=true \
+  --set hubble.ui.enabled=true \
+  --set hubble.metrics.enabled="{dns,drop,tcp,flow,port-distribution,icmp,httpV2:exemplars=true;labelsContext=source_ip\,source_namespace\,source_workload\,destination_ip\,destination_namespace\,destination_workload\,traffic_direction}"
+```
+
+---
+
+## 🛡️ L7 CiliumNetworkPolicy Specification
+
+Enforce strict Zero-Trust boundaries allowing only authorized HTTP/gRPC methods between microservices:
+
+```yaml
+apiVersion: "cilium.io/v2"
+kind: CiliumNetworkPolicy
+metadata:
+  name: order-service-l7-policy
+  namespace: production
+spec:
+  endpointSelector:
+    matchLabels:
+      app: order-service
+  ingress:
+    - fromEndpoints:
+        - matchLabels:
+            app: api-gateway
+      toPorts:
+        - ports:
+            - port: "8080"
+              protocol: TCP
+          rules:
+            http:
+              - method: "POST"
+                path: "/v1/orders"
+              - method: "GET"
+                path: "/v1/orders/.*"
+  egress:
+    - toEndpoints:
+        - matchLabels:
+            app: payment-service
+      toPorts:
+        - ports:
+            - port: "8443"
+              protocol: TCP
+          rules:
+            http:
+              - method: "POST"
+                path: "/v1/charges"
+    - toFQDNs:
+        - matchName: "api.stripe.com"
+      toPorts:
+        - ports:
+            - port: "443"
+              protocol: TCP
+```
+
+---
+
+## 🔍 Hubble CLI Diagnostic Workflows
+
+The Hubble CLI provides real-time kernel packet inspection:
+
+### 1. Tail Dropped Packets in Namespace
+
+```bash
+hubble observe --namespace production --verdict DROP -f
+```
+
+### 2. Trace HTTP Latency & 5xx Status Errors
+
+```bash
+hubble observe \
+  --namespace production \
+  --protocol http \
+  --http-status 5.. \
+  --output jsonpb | jq '{source: .flow.source.pod_name, dest: .flow.destination.pod_name, status: .flow.l7.http.code}'
+```
+
+### 3. Inspect DNS Queries and Lookup Failures
+
+```bash
+hubble observe --namespace production --protocol dns
+```
+
+---
+
+## 📈 Prometheus Service Metrics Integration
+
+Cilium exposes Prometheus metrics on port `9962` (agent) and `9965` (operator):
+
+```text
+cilium_forward_count_total       # Total packets forwarded through eBPF datapath
+cilium_drop_count_total          # Total packets dropped by policy or checksum failure
+cilium_bpf_map_pressure          # Map utilization (target: < 0.70 to avoid overflow)
+hubble_flows_processed_total     # Flow event processing rate
+```
+
+---
+
+## ⚠️ Troubleshooting & Production Checklist
+
+1. **BPF Map Pressure Alert**:
+   - Inspect active map sizing: `cilium-dbg bpf map list`.
+   - If `cilium_lb4_services_v2` exceeds 80%, increase `--set bpf.lbMapMax=65536`.
+2. **WireGuard Encryption Verification**:
+   - Run `cilium status --verbose` and confirm `Encryption: WireGuard [Node: OK, Pod: OK]`.
+3. **Kernel Version Compatibility**:
+   - Requires Linux Kernel >= 5.10 for full kube-proxy replacement, and >= 5.15 for transparent mTLS encryption.

package/skills/clickhouse-analytics/SKILL.md

@@ -0,0 +1,128 @@
+---
+name: clickhouse-analytics
+description: >-
+  Architects, models, and optimizes high-throughput ClickHouse OLAP analytical schemas, MergeTree table engines, real-time materialized views, and vector similarity search.
+---
+
+# ClickHouse Analytics & Real-Time OLAP
+
+This skill guides high-throughput schema modeling, query optimization, and real-time streaming ingestion on ClickHouse for analytics and vector retrieval.
+
+---
+
+## ⚡ Core Engine & Table Design
+
+### 1. Engine Selection Matrix
+
+| Engine | Best For | Behavior |
+| :--- | :--- | :--- |
+| **`MergeTree`** | Default append-only timeseries | Base engine for fast parallel range scans and primary indexing. |
+| **`ReplacingMergeTree`** | Deduplication & upserts | Replaces rows with matching sorting keys during background compaction. |
+| **`SummingMergeTree`** | Real-time counter rollups | Automatically aggregates numeric columns with identical sorting keys. |
+| **`AggregatingMergeTree`**| Pre-aggregated state metrics | Stores intermediate aggregate states (e.g. `uniqExactState`, `quantilesState`). |
+
+---
+
+### 2. Primary Key & Partitioning Strategy
+
+In ClickHouse, the `ORDER BY` defines the sparse primary index (1 index mark every 8,192 rows). Order columns by ascending cardinality:
+
+```sql
+CREATE TABLE analytics.events (
+    tenant_id UUID,
+    event_type LowCardinality(String),
+    user_id String,
+    properties JSON,
+    timestamp DateTime64(3, 'UTC')
+) ENGINE = MergeTree()
+PARTITION BY toYYYYMM(timestamp)
+ORDER BY (tenant_id, event_type, timestamp, user_id)
+SETTINGS index_granularity = 8192, ttl_only_drop_parts = 1;
+```
+
+> **Rules:**
+> 1. Never partition by high-cardinality fields (like `user_id` or `toDate(timestamp)`); keep partition counts under 1,000 to prevent metadata explosion.
+> 2. Use `LowCardinality(String)` for strings with < 10,000 distinct values (e.g., country codes, event names) for dictionary encoding.
+
+---
+
+## 🔄 Real-Time Ingestion with Kafka & Materialized Views
+
+Stream data continuously from Apache Kafka directly into optimized storage tables using pipe tables and materialized views:
+
+```sql
+-- 1. Kafka Queue Engine
+CREATE TABLE analytics.kafka_events_queue (
+    raw_payload String
+) ENGINE = Kafka
+SETTINGS kafka_broker_list = 'kafka:9092',
+         kafka_topic_list = 'app-events',
+         kafka_group_name = 'clickhouse-consumer',
+         kafka_format = 'JSONAsString';
+
+-- 2. Destination Storage Table
+CREATE TABLE analytics.events_store (
+    event_id UUID,
+    event_name LowCardinality(String),
+    payload JSON,
+    created_at DateTime DEFAULT now()
+) ENGINE = MergeTree()
+ORDER BY (event_name, created_at);
+
+-- 3. Materialized View Pipe
+CREATE MATERIALIZED VIEW analytics.mv_kafka_to_events TO analytics.events_store AS
+SELECT
+    JSONExtract(raw_payload, 'id', 'UUID') AS event_id,
+    JSONExtract(raw_payload, 'type', 'LowCardinality(String)') AS event_name,
+    raw_payload AS payload,
+    now() AS created_at
+FROM analytics.kafka_events_queue;
+```
+
+---
+
+## 🚀 High-Performance Query Optimization
+
+### 1. Leverage `PREWHERE` for Selective Filters
+`PREWHERE` reads filtered columns first and only fetches remaining requested columns for matching rows:
+
+```sql
+SELECT
+    user_id,
+    count() AS total_events,
+    quantile(0.95)(JSONExtractFloat(properties, 'latency_ms')) AS p95_latency
+FROM analytics.events
+PREWHERE tenant_id = 'a3bb189e-8bf9-4888-9912-abcdef012345'
+  AND timestamp >= now() - INTERVAL 24 HOUR
+WHERE event_type = 'api_request'
+GROUP BY user_id
+ORDER BY total_events DESC
+LIMIT 50;
+```
+
+### 2. Approximate Aggregations for Massive Scale
+- Use `uniqCombined64(user_id)` or `uniqHLL12(user_id)` instead of `uniqExact()` when calculating HyperLogLog DAU/MAU across hundreds of millions of events (~10x faster, <1% error).
+
+---
+
+## 🧠 Vector Similarity Search in ClickHouse
+
+ClickHouse natively computes high-dimensional vector distances:
+
+```sql
+CREATE TABLE analytics.article_embeddings (
+    id UUID,
+    title String,
+    embedding Array(Float32),
+    INDEX idx_emb embedding TYPE vector_similarity('hnsw', 'cosineDistance') GRANULARITY 1
+) ENGINE = MergeTree()
+ORDER BY id;
+
+-- Retrieve Top-K Nearest Neighbors
+SELECT
+    title,
+    cosineDistance(embedding, [0.021, -0.145, 0.891, ...]) AS score
+FROM analytics.article_embeddings
+ORDER BY score ASC
+LIMIT 10;
+```

package/skills/dbt-transformation/SKILL.md

@@ -0,0 +1,179 @@
+---
+name: dbt-transformation
+description: >-
+  Designs, develops, and tests modular SQL data transformations using dbt (data build tool), Jinja macros, semantic layer metrics, automated schema tests, and documentation generation.
+---
+
+# dbt Modular Transformation & Data Modeling
+
+This skill guides structuring production-grade ELT pipelines using dbt (Core & Cloud), enforcing modular layer architecture (staging, intermediate, marts), DRY Jinja macros, automated data testing, and incremental table materialization.
+
+---
+
+## ⚡ Core Patterns
+
+### 1. Multi-Layer Dimensional Modeling Architecture
+
+Organize transformations into clean, progressive DAG layers:
+
+```text
+models/
+├── staging/                     # 1:1 view on raw sources (renaming, casting)
+│   ├── stripe/
+│   │   ├── _stripe__sources.yml
+│   │   ├── _stripe__models.yml
+│   │   └── stg_stripe__payments.sql
+├── intermediate/                # Business logic joins & window aggregations
+│   └── finance/
+│       ├── _int_finance__models.yml
+│       └── int_payments_pivoted_to_orders.sql
+└── marts/                       # Dimensional star schema (facts & dimensions)
+    ├── core/
+    │   ├── fct_orders.sql
+    │   └── dim_customers.sql
+    └── marketing/
+        └── fct_campaign_conversions.sql
+```
+
+---
+
+### 2. Staging Layer Model with CTE Standards
+
+Clean raw source tables into normalized relational types:
+
+```sql
+-- models/staging/stripe/stg_stripe__payments.sql
+with source as (
+    select * from {{ source('stripe', 'payment') }}
+),
+
+renamed as (
+    select
+        id as payment_id,
+        orderid as order_id,
+        paymentmethod as payment_method,
+        status as payment_status,
+        -- Amount stored in cents; convert to dollars
+        round(amount / 100.0, 2) as amount_usd,
+        created as created_at
+    from source
+)
+
+select * from renamed
+```
+
+Declare tests and column descriptions in accompanying YAML:
+
+```yaml
+# models/staging/stripe/_stripe__models.yml
+version: 2
+
+models:
+  - name: stg_stripe__payments
+    description: "Normalized payments table from Stripe webhook ingest."
+    columns:
+      - name: payment_id
+        description: "Primary key for payment events."
+        tests:
+          - unique
+          - not_null
+      - name: payment_status
+        tests:
+          - accepted_values:
+              values: ['success', 'fail', 'pending']
+```
+
+---
+
+### 3. High-Performance Incremental Fact Table
+
+Process only newly ingested or updated records using `is_incremental()`:
+
+```sql
+-- models/marts/core/fct_orders.sql
+{{
+    config(
+        materialized='incremental',
+        unique_key='order_id',
+        incremental_strategy='merge',
+        on_schema_change='fail'
+    )
+}}
+
+with orders as (
+    select * from {{ ref('stg_jaffle_shop__orders') }}
+    {% if is_incremental() %}
+        -- Only query rows created or modified in the last 3 days for backfill resilience
+        where order_date >= (select dateadd('day', -3, max(order_date)) from {{ this }})
+    {% endif %}
+),
+
+payments as (
+    select * from {{ ref('stg_stripe__payments') }}
+),
+
+order_payments as (
+    select
+        order_id,
+        sum(case when payment_status = 'success' then amount_usd else 0 end) as amount_usd
+    from payments
+    group by 1
+),
+
+final as (
+    select
+        orders.order_id,
+        orders.customer_id,
+        orders.order_date,
+        orders.status as order_status,
+        coalesce(order_payments.amount_usd, 0) as total_amount_usd
+    from orders
+    left join order_payments on orders.order_id = order_payments.order_id
+)
+
+select * from final
+```
+
+---
+
+### 4. Custom Jinja Macro for Dynamic Pivoting
+
+Avoid repetitive boilerplate using parameterized Jinja macros:
+
+```sql
+-- macros/pivot_sum.sql
+{% macro pivot_sum(column_name, values_list, aggregate_col='amount_usd') %}
+    {% for val in values_list %}
+        sum(case when {{ column_name }} = '{{ val }}' then {{ aggregate_col }} else 0 end) as {{ val }}_amount
+        {% if not loop.last %},{% endif %}
+    {% endfor %}
+{% endmacro %}
+```
+
+Usage in intermediate model:
+
+```sql
+select
+    order_id,
+    {{ pivot_sum('payment_method', ['credit_card', 'coupon', 'bank_transfer', 'gift_card']) }}
+from {{ ref('stg_stripe__payments') }}
+group by 1
+```
+
+---
+
+## 🎯 Verification & Diagnostics
+
+1. **Compile SQL with Resolved Jinja**:
+   ```bash
+   dbt compile --select fct_orders
+   ```
+2. **Execute Targeted Subgraph**:
+   ```bash
+   dbt run --select +fct_orders       # Run fct_orders and all upstream dependencies
+   dbt test --select stg_stripe+      # Test staging models and downstream marts
+   ```
+3. **Build Documentation & DAG Visualizer**:
+   ```bash
+   dbt docs generate && dbt docs serve
+   ```