everything-gemini 1.2.0 → 1.9.0

package/skills/argo-workflows/SKILL.md

@@ -0,0 +1,151 @@
+---
+name: argo-workflows
+description: >-
+  Architects, defines, and executes Kubernetes-native DAG workflows, artifact passing, container steps, retry strategies, and CronWorkflows using Argo Workflows.
+---
+
+# Argo Workflows & Kubernetes-Native DAG Orchestration
+
+This skill provides operational patterns for designing, testing, and deploying Kubernetes-native workflow orchestration via Argo Workflows, powering parallel data pipelines, machine learning training graphs, and automated batch processing.
+
+---
+
+## ⚡ Core Architecture & CRDs
+
+1. **`Workflow`**: Ephemeral, single-run execution specification.
+2. **`WorkflowTemplate`**: Reusable workflow definitions stored inside the Kubernetes cluster.
+3. **`CronWorkflow`**: Time-triggered recurring workflows (cron schedule).
+4. **DAGs vs Steps**: DAGs declare declarative dependency graphs (`dependencies: [taskA, taskB]`), whereas Steps declare sequential matrix execution.
+5. **Artifacts**: Input/output binary blobs, model weights, and logs backed by S3, GCS, or MinIO.
+
+---
+
+## 🏗️ Production DAG Workflow Specification (`workflow-dag.yaml`)
+
+```yaml
+apiVersion: argoproj.io/v1alpha1
+kind: Workflow
+metadata:
+  generateName: ml-pipeline-dag-
+  namespace: argo
+spec:
+  entrypoint: ml-pipeline
+  serviceAccountName: argo-workflow-executor
+  shutdown: "Stop" # Graceful termination support
+  onExit: pipeline-cleanup-handler
+
+  templates:
+    - name: ml-pipeline
+      dag:
+        tasks:
+          - name: ingest-data
+            template: data-ingestion-step
+            arguments:
+              parameters:
+                - name: dataset-version
+                  value: "v2026.10"
+
+          - name: validate-schema
+            dependencies: [ingest-data]
+            template: schema-validator-step
+            arguments:
+              artifacts:
+                - name: raw-dataset
+                  from: "{{tasks.ingest-data.outputs.artifacts.dataset}}"
+
+          - name: train-model
+            dependencies: [validate-schema]
+            template: model-training-step
+            arguments:
+              artifacts:
+                - name: validated-data
+                  from: "{{tasks.validate-schema.outputs.artifacts.clean-dataset}}"
+
+    - name: data-ingestion-step
+      inputs:
+        parameters:
+          - name: dataset-version
+      container:
+        image: python:3.11-slim
+        command: [python, -c]
+        args: ["import json; open('/tmp/dataset.csv', 'w').write('id,score\\n1,0.95\\n')"]
+      outputs:
+        artifacts:
+          - name: dataset
+            path: /tmp/dataset.csv
+
+    - name: schema-validator-step
+      inputs:
+        artifacts:
+          - name: raw-dataset
+            path: /inputs/dataset.csv
+      container:
+        image: python:3.11-slim
+        command: [python, -c]
+        args: ["import shutil; shutil.copy('/inputs/dataset.csv', '/outputs/clean.csv')"]
+      outputs:
+        artifacts:
+          - name: clean-dataset
+            path: /outputs/clean.csv
+
+    - name: model-training-step
+      inputs:
+        artifacts:
+          - name: validated-data
+            path: /data/clean.csv
+      retryStrategy:
+        limit: "3"
+        retryPolicy: "Always"
+        backoff:
+          duration: "10s"
+          factor: 2
+      container:
+        image: python:3.11-slim
+        resources:
+          requests:
+            cpu: "2"
+            memory: "4Gi"
+          limits:
+            cpu: "4"
+            memory: "8Gi"
+        command: [python, -c]
+        args: ["print('Model trained successfully')"]
+
+    - name: pipeline-cleanup-handler
+      container:
+        image: alpine:latest
+        command: [sh, -c]
+        args: ["echo Pipeline finished with status: {{workflow.status}}"]
+```
+
+---
+
+## 🛠️ CLI Operations & Debugging Runbook
+
+```bash
+# Submit a workflow and follow real-time execution
+argo submit --watch workflow-dag.yaml -n argo
+
+# List active and completed workflows
+argo list -n argo
+
+# Stream logs of a specific DAG task step
+argo logs @latest -c main -n argo
+
+# Resubmit a failed workflow with memoized step caching
+argo resubmit --memoized @latest -n argo
+
+# Terminate or suspend a runaway workflow
+argo terminate @latest -n argo
+```
+
+---
+
+## 🛡️ Best Practices & Anti-Patterns
+
+| Category | ❌ Anti-Pattern | ✅ Production Best Practice |
+| :--- | :--- | :--- |
+| **Artifacts** | Storing giant datasets directly inside pod disk | Use external artifact repositories (S3 / GCS) with lifecycle policies |
+| **Retries** | Unbounded retries without exponential backoff | Set explicit `retryStrategy.limit` and exponential `backoff.factor` |
+| **Resources** | Omitting CPU/Memory resource constraints | Always specify `requests` and `limits` to prevent node OOM crashes |
+| **Secrets** | Hardcoding API keys or database URLs in YAML | Reference Kubernetes Secrets via `env.valueFrom.secretKeyRef` |

package/skills/axum-web-service/SKILL.md

@@ -0,0 +1,159 @@
+---
+name: axum-web-service
+description: >-
+  Architects, builds, and optimizes production Rust web APIs using Axum, Tokio async runtime, Tower middlewares, type-safe state extractors, and SQLx connection pools.
+---
+
+# Axum Web Service Architecture (Rust)
+
+This skill guides engineering microservices with Axum—the high-throughput, ergonomic Rust web framework built on top of Tokio, Tower, and Hyper.
+
+---
+
+## ⚡ Core Patterns
+
+### 1. Application State & Router Composition (`main.rs`)
+
+```rust
+use axum::{
+    extract::{Path, State},
+    http::StatusCode,
+    response::IntoResponse,
+    routing::{get, post},
+    Json, Router,
+};
+use sqlx::PgPool;
+use std::{net::SocketAddr, sync::Arc};
+use tower_http::{cors::CorsLayer, trace::TraceLayer};
+
+#[derive(Clone)]
+pub struct AppState {
+    pub db: PgPool,
+    pub redis: redis::aio::ConnectionManager,
+}
+
+#[tokio::main]
+async fn main() -> Result<(), Box<dyn std::error::Error>> {
+    tracing_subscriber::fmt::init();
+
+    let db_pool = PgPool::connect(&std::env::var("DATABASE_URL")?).await?;
+    let redis_client = redis::Client::open(std::env::var("REDIS_URL")?)?;
+    let redis_mgr = redis_client.get_tokio_connection_manager().await?;
+
+    let shared_state = Arc::new(AppState {
+        db: db_pool,
+        redis: redis_mgr,
+    });
+
+    let app = Router::new()
+        .route("/healthz", get(|| async { StatusCode::OK }))
+        .nest("/api/v1", api_v1_router())
+        .layer(TraceLayer::new_for_http())
+        .layer(CorsLayer::permissive())
+        .with_state(shared_state);
+
+    let addr = SocketAddr::from(([0, 0, 0, 0], 8080));
+    let listener = tokio::net::TcpListener::bind(addr).await?;
+    tracing::info!("Listening on {}", addr);
+    axum::serve(listener, app).await?;
+
+    Ok(())
+}
+```
+
+---
+
+### 2. Type-Safe Handlers with SQLx (`handlers/users.rs`)
+
+Use Axum's compile-time extractors to ensure payload and state invariants before handler execution:
+
+```rust
+use axum::{
+    extract::{Path, State},
+    Json,
+};
+use serde::{Deserialize, Serialize};
+use std::sync::Arc;
+use crate::{AppState, error::AppError};
+
+#[derive(Deserialize)]
+pub struct CreateUserPayload {
+    pub username: String,
+    pub email: String,
+}
+
+#[derive(Serialize)]
+pub struct UserResponse {
+    pub id: i64,
+    pub username: String,
+    pub email: String,
+}
+
+pub async fn create_user(
+    State(state): State<Arc<AppState>>,
+    Json(payload): Json<CreateUserPayload>,
+) -> Result<Json<UserResponse>, AppError> {
+    // Compile-time verified query via SQLx
+    let user = sqlx::query_as!(
+        UserResponse,
+        r#"
+        INSERT INTO users (username, email)
+        VALUES ($1, $2)
+        RETURNING id, username, email
+        "#,
+        payload.username,
+        payload.email
+    )
+    .fetch_one(&state.db)
+    .await?;
+
+    Ok(Json(user))
+}
+```
+
+---
+
+### 3. Unified RFC 7807 Error Envelope (`error.rs`)
+
+Transform any internal database or validation failure into clean, typed JSON responses using `IntoResponse`:
+
+```rust
+use axum::{
+    http::StatusCode,
+    response::{IntoResponse, Response},
+    Json,
+};
+use serde_json::json;
+
+#[derive(thiserror::Error, Debug)]
+pub enum AppError {
+    #[error("Database error: {0}")]
+    Database(#[from] sqlx::Error),
+    #[error("Resource not found")]
+    NotFound,
+    #[error("Validation failed: {0}")]
+    Validation(String),
+}
+
+impl IntoResponse for AppError {
+    fn into_response(self) -> Response {
+        let (status, detail) = match self {
+            AppError::NotFound => (StatusCode::NOT_FOUND, "The requested resource does not exist.".to_string()),
+            AppError::Validation(msg) => (StatusCode::BAD_REQUEST, msg),
+            AppError::Database(err) => {
+                tracing::error!("Internal DB error: {:?}", err);
+                (StatusCode::INTERNAL_SERVER_ERROR, "An internal server error occurred.".to_string())
+            }
+        };
+
+        let body = Json(json!({
+            "type": "https://errors.example.com",
+            "title": status.canonical_reason().unwrap_or("Error"),
+            "status": status.as_u16(),
+            "detail": detail
+        }));
+
+        (status, body).into_response()
+    }
+}
+```

package/skills/cilium-ebpf-networking/SKILL.md

@@ -0,0 +1,145 @@
+---
+name: cilium-ebpf-networking
+description: >-
+  Cloud-native eBPF networking, Kubernetes CNI replacement, transparent mutual TLS (mTLS), L7 CiliumNetworkPolicies, and Hubble network observability. Use when securing cluster traffic, eliminating kube-proxy iptables bottlenecks, or debugging service mesh latency.
+---
+
+# Cilium eBPF Networking & Security Architecture
+
+A production runbook for deploying, configuring, and troubleshooting **Cilium eBPF**, high-performance Kubernetes networking, and deep packet observability with **Hubble**.
+
+---
+
+## 🎯 Architectural Advantages of eBPF
+
+1. **Bypasses iptables / IPVS Overhead**: Programs run directly in the Linux kernel network stack (`tc` / `xdp`), providing O(1) connection routing regardless of service count.
+2. **Kernel-Level Identity**: Enforces security policies based on cryptographically verified Kubernetes metadata and SPIFFE identities rather than volatile IP addresses.
+3. **Transparent Encryption**: Mutual TLS (mTLS) or WireGuard encryption negotiated dynamically at the kernel layer with zero application sidecars.
+4. **Hubble Observability**: Zero-overhead flow logging, DNS resolution tracking, and L7 HTTP/gRPC metrics without code instrumentation.
+
+---
+
+## 🚀 Helm Deployment (kube-proxy Replacement)
+
+```bash
+helm repo add cilium https://helm.cilium.io/
+helm repo update
+
+helm install cilium cilium/cilium --version 1.16.0 \
+  --namespace kube-system \
+  --set kubeProxyReplacement=true \
+  --set k8sServiceHost=10.0.0.1 \
+  --set k8sServicePort=6443 \
+  --set routingMode=native \
+  --set autoDirectNodeRoutes=true \
+  --set bpf.masquerade=true \
+  --set encryption.enabled=true \
+  --set encryption.type=wireguard \
+  --set hubble.enabled=true \
+  --set hubble.relay.enabled=true \
+  --set hubble.ui.enabled=true \
+  --set hubble.metrics.enabled="{dns,drop,tcp,flow,port-distribution,icmp,httpV2:exemplars=true;labelsContext=source_ip\,source_namespace\,source_workload\,destination_ip\,destination_namespace\,destination_workload\,traffic_direction}"
+```
+
+---
+
+## 🛡️ L7 CiliumNetworkPolicy Specification
+
+Enforce strict Zero-Trust boundaries allowing only authorized HTTP/gRPC methods between microservices:
+
+```yaml
+apiVersion: "cilium.io/v2"
+kind: CiliumNetworkPolicy
+metadata:
+  name: order-service-l7-policy
+  namespace: production
+spec:
+  endpointSelector:
+    matchLabels:
+      app: order-service
+  ingress:
+    - fromEndpoints:
+        - matchLabels:
+            app: api-gateway
+      toPorts:
+        - ports:
+            - port: "8080"
+              protocol: TCP
+          rules:
+            http:
+              - method: "POST"
+                path: "/v1/orders"
+              - method: "GET"
+                path: "/v1/orders/.*"
+  egress:
+    - toEndpoints:
+        - matchLabels:
+            app: payment-service
+      toPorts:
+        - ports:
+            - port: "8443"
+              protocol: TCP
+          rules:
+            http:
+              - method: "POST"
+                path: "/v1/charges"
+    - toFQDNs:
+        - matchName: "api.stripe.com"
+      toPorts:
+        - ports:
+            - port: "443"
+              protocol: TCP
+```
+
+---
+
+## 🔍 Hubble CLI Diagnostic Workflows
+
+The Hubble CLI provides real-time kernel packet inspection:
+
+### 1. Tail Dropped Packets in Namespace
+
+```bash
+hubble observe --namespace production --verdict DROP -f
+```
+
+### 2. Trace HTTP Latency & 5xx Status Errors
+
+```bash
+hubble observe \
+  --namespace production \
+  --protocol http \
+  --http-status 5.. \
+  --output jsonpb | jq '{source: .flow.source.pod_name, dest: .flow.destination.pod_name, status: .flow.l7.http.code}'
+```
+
+### 3. Inspect DNS Queries and Lookup Failures
+
+```bash
+hubble observe --namespace production --protocol dns
+```
+
+---
+
+## 📈 Prometheus Service Metrics Integration
+
+Cilium exposes Prometheus metrics on port `9962` (agent) and `9965` (operator):
+
+```text
+cilium_forward_count_total       # Total packets forwarded through eBPF datapath
+cilium_drop_count_total          # Total packets dropped by policy or checksum failure
+cilium_bpf_map_pressure          # Map utilization (target: < 0.70 to avoid overflow)
+hubble_flows_processed_total     # Flow event processing rate
+```
+
+---
+
+## ⚠️ Troubleshooting & Production Checklist
+
+1. **BPF Map Pressure Alert**:
+   - Inspect active map sizing: `cilium-dbg bpf map list`.
+   - If `cilium_lb4_services_v2` exceeds 80%, increase `--set bpf.lbMapMax=65536`.
+2. **WireGuard Encryption Verification**:
+   - Run `cilium status --verbose` and confirm `Encryption: WireGuard [Node: OK, Pod: OK]`.
+3. **Kernel Version Compatibility**:
+   - Requires Linux Kernel >= 5.10 for full kube-proxy replacement, and >= 5.15 for transparent mTLS encryption.

package/skills/clickhouse-analytics/SKILL.md

@@ -0,0 +1,128 @@
+---
+name: clickhouse-analytics
+description: >-
+  Architects, models, and optimizes high-throughput ClickHouse OLAP analytical schemas, MergeTree table engines, real-time materialized views, and vector similarity search.
+---
+
+# ClickHouse Analytics & Real-Time OLAP
+
+This skill guides high-throughput schema modeling, query optimization, and real-time streaming ingestion on ClickHouse for analytics and vector retrieval.
+
+---
+
+## ⚡ Core Engine & Table Design
+
+### 1. Engine Selection Matrix
+
+| Engine | Best For | Behavior |
+| :--- | :--- | :--- |
+| **`MergeTree`** | Default append-only timeseries | Base engine for fast parallel range scans and primary indexing. |
+| **`ReplacingMergeTree`** | Deduplication & upserts | Replaces rows with matching sorting keys during background compaction. |
+| **`SummingMergeTree`** | Real-time counter rollups | Automatically aggregates numeric columns with identical sorting keys. |
+| **`AggregatingMergeTree`**| Pre-aggregated state metrics | Stores intermediate aggregate states (e.g. `uniqExactState`, `quantilesState`). |
+
+---
+
+### 2. Primary Key & Partitioning Strategy
+
+In ClickHouse, the `ORDER BY` defines the sparse primary index (1 index mark every 8,192 rows). Order columns by ascending cardinality:
+
+```sql
+CREATE TABLE analytics.events (
+    tenant_id UUID,
+    event_type LowCardinality(String),
+    user_id String,
+    properties JSON,
+    timestamp DateTime64(3, 'UTC')
+) ENGINE = MergeTree()
+PARTITION BY toYYYYMM(timestamp)
+ORDER BY (tenant_id, event_type, timestamp, user_id)
+SETTINGS index_granularity = 8192, ttl_only_drop_parts = 1;
+```
+
+> **Rules:**
+> 1. Never partition by high-cardinality fields (like `user_id` or `toDate(timestamp)`); keep partition counts under 1,000 to prevent metadata explosion.
+> 2. Use `LowCardinality(String)` for strings with < 10,000 distinct values (e.g., country codes, event names) for dictionary encoding.
+
+---
+
+## 🔄 Real-Time Ingestion with Kafka & Materialized Views
+
+Stream data continuously from Apache Kafka directly into optimized storage tables using pipe tables and materialized views:
+
+```sql
+-- 1. Kafka Queue Engine
+CREATE TABLE analytics.kafka_events_queue (
+    raw_payload String
+) ENGINE = Kafka
+SETTINGS kafka_broker_list = 'kafka:9092',
+         kafka_topic_list = 'app-events',
+         kafka_group_name = 'clickhouse-consumer',
+         kafka_format = 'JSONAsString';
+
+-- 2. Destination Storage Table
+CREATE TABLE analytics.events_store (
+    event_id UUID,
+    event_name LowCardinality(String),
+    payload JSON,
+    created_at DateTime DEFAULT now()
+) ENGINE = MergeTree()
+ORDER BY (event_name, created_at);
+
+-- 3. Materialized View Pipe
+CREATE MATERIALIZED VIEW analytics.mv_kafka_to_events TO analytics.events_store AS
+SELECT
+    JSONExtract(raw_payload, 'id', 'UUID') AS event_id,
+    JSONExtract(raw_payload, 'type', 'LowCardinality(String)') AS event_name,
+    raw_payload AS payload,
+    now() AS created_at
+FROM analytics.kafka_events_queue;
+```
+
+---
+
+## 🚀 High-Performance Query Optimization
+
+### 1. Leverage `PREWHERE` for Selective Filters
+`PREWHERE` reads filtered columns first and only fetches remaining requested columns for matching rows:
+
+```sql
+SELECT
+    user_id,
+    count() AS total_events,
+    quantile(0.95)(JSONExtractFloat(properties, 'latency_ms')) AS p95_latency
+FROM analytics.events
+PREWHERE tenant_id = 'a3bb189e-8bf9-4888-9912-abcdef012345'
+  AND timestamp >= now() - INTERVAL 24 HOUR
+WHERE event_type = 'api_request'
+GROUP BY user_id
+ORDER BY total_events DESC
+LIMIT 50;
+```
+
+### 2. Approximate Aggregations for Massive Scale
+- Use `uniqCombined64(user_id)` or `uniqHLL12(user_id)` instead of `uniqExact()` when calculating HyperLogLog DAU/MAU across hundreds of millions of events (~10x faster, <1% error).
+
+---
+
+## 🧠 Vector Similarity Search in ClickHouse
+
+ClickHouse natively computes high-dimensional vector distances:
+
+```sql
+CREATE TABLE analytics.article_embeddings (
+    id UUID,
+    title String,
+    embedding Array(Float32),
+    INDEX idx_emb embedding TYPE vector_similarity('hnsw', 'cosineDistance') GRANULARITY 1
+) ENGINE = MergeTree()
+ORDER BY id;
+
+-- Retrieve Top-K Nearest Neighbors
+SELECT
+    title,
+    cosineDistance(embedding, [0.021, -0.145, 0.891, ...]) AS score
+FROM analytics.article_embeddings
+ORDER BY score ASC
+LIMIT 10;
+```