everything-gemini 1.9.0 → 2.1.0

package/skills/chromadb-vector-store/SKILL.md

@@ -0,0 +1,188 @@
+---
+name: chromadb-vector-store
+description: Implements ChromaDB persistent vector storage, multi-modal embeddings, collection partitioning, metadata filtering, and semantic similarity search.
+---
+
+# ChromaDB Vector Store
+
+Provides production patterns for embedding generation, persistent collection management, metadata-filtered semantic retrieval, and hybrid search using ChromaDB.
+
+---
+
+## 1. Client Architecture
+
+ChromaDB operates in three deployment modes depending on scale and latency requirements:
+
+```
+  In-Process Ephemeral            Local Persistent                Client / Server
+┌──────────────────────┐     ┌───────────────────────┐     ┌────────────────────────┐
+│ chromadb.Client()    │     │ PersistentClient(path)│     │ HttpClient(host, port) │
+│ (Unit Tests & CI)    │     │ (Embedded SQLite/HNSW)│     │ (Kubernetes / Docker)  │
+└──────────────────────┘     └───────────────────────┘     └────────────────────────┘
+```
+
+```python
+import chromadb
+from chromadb.config import Settings
+
+# Production persistent client with DuckDB + HNSW backend
+client = chromadb.PersistentClient(
+    path="./data/chroma",
+    settings=Settings(
+        anonymized_telemetry=False,
+        allow_reset=False,
+    )
+)
+```
+
+---
+
+## 2. Gemini Embeddings & Collection Creation
+
+Integrates native Gemini embeddings (`text-embedding-004`) with custom collection metrics:
+
+```python
+import chromadb
+from chromadb.api.types import Documents, EmbeddingFunction, Embeddings
+from google import genai
+
+class GeminiEmbeddingFunction(EmbeddingFunction):
+    def __init__(self, model_name: str = "text-embedding-004"):
+        self.client = genai.Client()
+        self.model_name = model_name
+
+    def __call__(self, input: Documents) -> Embeddings:
+        response = self.client.models.embed_content(
+            model=self.model_name,
+            contents=input,
+        )
+        return [e.values for e in response.embeddings]
+
+gemini_ef = GeminiEmbeddingFunction()
+
+# Create or get collection with Cosine distance metric
+collection = client.get_or_create_collection(
+    name="technical_runbooks_v2",
+    embedding_function=gemini_ef,
+    metadata={"hnsw:space": "cosine", "description": "Antigravity skills catalog"}
+)
+```
+
+---
+
+## 3. High-Throughput Batch Upsert
+
+Batch insertion ensures zero duplicate embeddings and enforces metadata schemas:
+
+```python
+from typing import List, Dict, Any
+
+def upsert_knowledge_batch(
+    collection,
+    doc_ids: List[str],
+    texts: List[str],
+    metadatas: List[Dict[str, Any]]
+):
+    """Upserts documents in batches of 500 to stay within gRPC frame limits."""
+    batch_size = 500
+    for i in range(0, len(doc_ids), batch_size):
+        chunk_ids = doc_ids[i:i + batch_size]
+        chunk_texts = texts[i:i + batch_size]
+        chunk_metas = metadatas[i:i + batch_size]
+
+        collection.upsert(
+            ids=chunk_ids,
+            documents=chunk_texts,
+            metadatas=chunk_metas
+        )
+```
+
+---
+
+## 4. Metadata Filtering & Semantic Search
+
+Queries can filter across Boolean operators (`$and`, `$or`), numerical ranges, and substring matching:
+
+```python
+results = collection.query(
+    query_texts=["How do I configure KEDA ScaledObjects for Kafka?"],
+    n_results=5,
+    where={
+        "$and": [
+            {"category": {"$eq": "devops"}},
+            {"security_tier": {"$lte": 2}},
+            {"status": {"$ne": "deprecated"}}
+        ]
+    },
+    where_document={"$contains": "ScaledObject"},
+    include=["documents", "metadatas", "distances"]
+)
+
+# Parse matched documents
+for idx, (doc, meta, dist) in enumerate(zip(
+    results["documents"][0],
+    results["metadatas"][0],
+    results["distances"][0]
+)):
+    similarity_score = 1.0 - dist
+    print(f"[{idx+1}] Score: {similarity_score:.4f} | ID: {results['ids'][0][idx]}")
+    print(f"     Title: {meta.get('title')} (v{meta.get('version')})")
+```
+
+---
+
+## 5. Dockerized Server Deployment
+
+For multi-tenant or distributed Antigravity agents:
+
+```yaml
+# compose.chroma.yml
+version: "3.9"
+services:
+  chromadb:
+    image: chromadb/chroma:0.5.23
+    ports:
+      - "8000:8000"
+    environment:
+      - IS_PERSISTENT=TRUE
+      - PERSISTENCE_DIRECTORY=/chroma/chroma
+      - ANONYMIZED_TELEMETRY=FALSE
+    volumes:
+      - chroma_data:/chroma/chroma
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://localhost:8000/api/v1/heartbeat"]
+      interval: 10s
+      timeout: 5s
+      retries: 3
+
+volumes:
+  chroma_data:
+```
+
+```python
+# Connect via HTTP
+remote_client = chromadb.HttpClient(
+    host="localhost",
+    port=8000,
+    headers={"X-Chroma-Token": "secret-auth-token"}
+)
+```
+
+---
+
+## 6. Verification and Maintenance
+
+```bash
+# Verify Chroma API heartbeat
+curl -s http://localhost:8000/api/v1/heartbeat | jq .
+
+# Check active collections count and schema
+curl -s http://localhost:8000/api/v1/collections | jq '.[].name'
+
+# Reset collection or clean expired embeddings safely
+python -c "
+import chromadb
+client = chromadb.PersistentClient('./data/chroma')
+print(f'Collections: {len(client.list_collections())}')
+"
+```

package/skills/dataproc-spark-analytics/SKILL.md

@@ -0,0 +1,120 @@
+---
+name: dataproc-spark-analytics
+description: Deploys and tunes Google Cloud Dataproc Serverless Spark, PySpark batch pipelines, BigLake vector indexing, and Data Agent Kit telemetry.
+---
+
+# Dataproc Spark Analytics
+
+Orchestrates large-scale distributed data processing, feature engineering, and high-throughput vector embedding generation using Google Cloud Dataproc Serverless Spark and PySpark.
+
+---
+
+## 1. Architecture: Dataproc Serverless & Agentic Big Data
+
+```
+ Google Cloud Storage / BigLake         Dataproc Serverless               Gemini Vector Index
+┌─────────────────────────────┐     ┌────────────────────────┐     ┌────────────────────────┐
+│ Raw Document Lakes          │────>│ Serverless PySpark     │────>│ BigQuery Vector Search │
+│ (Parquet, JSON, Delta/Ice)  │     │ Batch Executors        │     │ / Milvus / ChromaDB    │
+└─────────────────────────────┘     └───────────┬────────────┘     └────────────────────────┘
+                                                │
+                                    ┌───────────▼────────────┐
+                                    │ Gemini Embedding API   │
+                                    │ (text-embedding-004)   │
+                                    └────────────────────────┘
+```
+
+- **Zero Cluster Management**: Runs ad-hoc PySpark workloads without provisioning or maintaining dedicated VMs.
+- **Dynamic Autoscaling**: Scales Spark executors elastically based on partition backlog and shuffle pressure.
+- **Unified Governance**: Native IAM Workload Identity federation and BigLake fine-grained column/row security.
+
+---
+
+## 2. Distributed Vector Embedding Pipeline (PySpark)
+
+Generates 768-dimensional embeddings over millions of technical documents using PySpark pandas UDFs:
+
+```python
+from pyspark.sql import SparkSession
+from pyspark.sql.functions import pandas_udf
+from pyspark.sql.types import ArrayType, FloatType
+import pandas as pd
+from typing import Iterator
+
+spark = SparkSession.builder \
+    .appName("Antigravity-Distributed-Embeddings") \
+    .config("spark.sql.execution.arrow.pyspark.enabled", "true") \
+    .getOrCreate()
+
+# Read partitioned documents from Cloud Storage
+docs_df = spark.read.parquet("gs://antigravity-data-lake/documents/v2/*.parquet")
+
+@pandas_udf(ArrayType(FloatType()))
+def compute_gemini_embeddings_udf(text_series: pd.Series) -> pd.Series:
+    """Computes embeddings in vectorized batches using official Google GenAI SDK."""
+    from google import genai
+    client = genai.Client()
+
+    results = []
+    # Batch requests in 100 items per API call
+    batch_size = 100
+    for i in range(0, len(text_series), batch_size):
+        chunk = text_series.iloc[i:i + batch_size].tolist()
+        resp = client.models.embed_content(
+            model="text-embedding-004",
+            contents=chunk
+        )
+        results.extend([e.values for e in resp.embeddings])
+    return pd.Series(results)
+
+# Transform and write to BigLake table
+embedded_df = docs_df.withColumn("embedding", compute_gemini_embeddings_udf("content"))
+embedded_df.write \
+    .format("bigquery") \
+    .mode("overwrite") \
+    .option("table", "enterprise_rag.document_embeddings_v2") \
+    .save()
+```
+
+---
+
+## 3. Submitting Serverless PySpark Batches
+
+Submits serverless batch jobs via Google Cloud CLI:
+
+```bash
+gcloud dataproc batches submit pyspark gs://antigravity-artifacts/scripts/generate_embeddings.py \
+  --project=my-gcp-ai-project \
+  --region=us-central1 \
+  --batch=batch-emb-$(date +%s) \
+  --deps-bucket=gs://antigravity-artifacts/staging \
+  --properties=spark.dynamicAllocation.maxExecutors=30 \
+  --version=2.2
+```
+
+---
+
+## 4. PySpark Performance Tuning Rules
+
+1. **Avoid `collect()`**: Never pull multi-gigabyte DataFrames onto the driver. Use `.write` directly to BigQuery or GCS.
+2. **Optimize Partition Sizing**: Aim for 128 MB – 256 MB per Spark partition:
+   ```python
+   df = df.repartition(100, "category_id")
+   ```
+3. **Broadcast Small Reference Maps**:
+   ```python
+   from pyspark.sql.functions import broadcast
+   joined_df = large_fact_df.join(broadcast(small_metadata_df), "tenant_id")
+   ```
+
+---
+
+## 5. Verification and Monitoring
+
+```bash
+# Check status of submitted batch
+gcloud dataproc batches describe <BATCH_ID> --region=us-central1 --format="value(state)"
+
+# Stream driver logs
+gcloud dataproc batches wait <BATCH_ID> --region=us-central1
+```

package/skills/google-flow-orchestrator/SKILL.md

@@ -0,0 +1,174 @@
+---
+name: google-flow-orchestrator
+description: Orchestrates Google Agentic Flow state machines, cyclic agent loops, human approval pauses, and multi-agent DAG execution graphs.
+---
+
+# Google Flow Orchestrator
+
+Architects and deploys deterministic, cyclic, and human-in-the-loop multi-agent workflows using Google Flow (Agentic Flow Engine) and Vertex AI Agent Builder.
+
+---
+
+## 1. Flow State Machine Architecture
+
+Google Flow models multi-agent execution as a directed, stateful graph where each node is an agent, tool invocation, or human checkpoint:
+
+```
+                     ┌──────────────────┐
+                     │   Start Node     │
+                     └────────┬─────────┘
+                              │
+                              ▼
+                     ┌──────────────────┐
+                     │ Planner Subagent │
+                     └────────┬─────────┘
+                              │
+                              ▼
+                     ┌──────────────────┐
+              ┌─────>│ Code Implementer│
+              │      └────────┬─────────┘
+              │               │
+     Failed   │               ▼
+   Verification      ┌──────────────────┐       Requires
+              ├──────│ Quality Auditor  │──────> Human Signoff
+              │      └────────┬─────────┘        (Pause State)
+              │               │ Passed                 │
+              │               ▼                        ▼
+              │      ┌──────────────────┐      ┌───────────────┐
+              └──────┤ Security Auditor │      │ Human Approval│
+                     └────────┬─────────┘      └───────┬───────┘
+                              │ Approved               │
+                              ▼                        ▼
+                     ┌─────────────────────────────────────────┐
+                     │          Deploy to Production           │
+                     └─────────────────────────────────────────┘
+```
+
+---
+
+## 2. Declarative Flow Definition (Python)
+
+Defines an immutable state graph with type-safe context, dynamic edge routing, and conditional transitions:
+
+```python
+from typing import Dict, Any, List, Optional
+from dataclasses import dataclass, field
+
+@dataclass
+class FlowState:
+    task_id: str
+    user_prompt: str
+    plan: Optional[str] = None
+    diff_patches: List[str] = field(default_factory=list)
+    test_results: Dict[str, Any] = field(default_factory=dict)
+    security_cleared: bool = False
+    human_approved: bool = False
+    iteration_count: int = 0
+    max_iterations: int = 5
+
+class AgenticFlow:
+    """Deterministic state machine coordinating Antigravity subagents."""
+
+    def __init__(self, initial_state: FlowState):
+        self.state = initial_state
+
+    def node_planner(self) -> str:
+        """Invokes Planner subagent to construct execution checklist."""
+        print(f"[FLOW:PLANNER] Creating roadmap for {self.state.task_id}...")
+        self.state.plan = "1. Add endpoint\n2. Run tests\n3. Audit security"
+        return "node_coder"
+
+    def node_coder(self) -> str:
+        """Applies code patches and increments iteration counter."""
+        self.state.iteration_count += 1
+        print(f"[FLOW:CODER] Generating patches (Iteration {self.state.iteration_count})...")
+        self.state.diff_patches.append("+ def handle_request(): pass")
+        return "node_tester"
+
+    def node_tester(self) -> str:
+        """Executes test verification gate."""
+        passed = self.state.iteration_count >= 1
+        self.state.test_results["suite_passed"] = passed
+        if not passed and self.state.iteration_count < self.state.max_iterations:
+            print("[FLOW:TESTER] Tests failed. Looping back to Coder.")
+            return "node_coder"
+        return "node_security"
+
+    def node_security(self) -> str:
+        """Evaluates OPA Gatekeeper and secret leakage."""
+        self.state.security_cleared = True
+        print("[FLOW:SECURITY] Security clearance verified.")
+        return "node_human_gate"
+
+    def node_human_gate(self) -> str:
+        """Halts execution for critical human-in-the-loop approval."""
+        if not self.state.human_approved:
+            print("[FLOW:HUMAN_GATE] Pausing workflow: Awaiting operator signoff.")
+            return "PAUSED"
+        return "node_release"
+
+    def node_release(self) -> str:
+        print("[FLOW:RELEASE] Production deployment completed.")
+        return "COMPLETED"
+```
+
+---
+
+## 3. Human-in-the-Loop Resumption
+
+Flows persist execution state into external stores (Redis / Cloud Spanner) to resume cleanly after human approval:
+
+```python
+def resume_flow_after_approval(flow: AgenticFlow, operator_id: str):
+    """Resumes paused flow once explicit signoff is received."""
+    print(f"✅ Received operator signoff from {operator_id}.")
+    flow.state.human_approved = True
+    next_step = flow.node_release()
+    print(f"Flow terminated with state: {next_step}")
+```
+
+---
+
+## 4. Integration with Google Cloud Vertex AI & Antigravity
+
+Maps Flow nodes directly to Antigravity's scoped subagents:
+
+```yaml
+apiVersion: flow.google.com/v1
+kind: AgentFlow
+metadata:
+  name: fullstack-feature-delivery
+spec:
+  entrypoint: planner
+  nodes:
+    - name: planner
+      agentRef: agents/planner.md
+      next: code-reviewer
+    - name: code-reviewer
+      agentRef: agents/code-reviewer.md
+      conditions:
+        - if: "diff_lines > 200"
+          then: security-auditor
+        - else: release-engineer
+    - name: security-auditor
+      agentRef: agents/security-auditor.md
+      next: release-engineer
+    - name: release-engineer
+      agentRef: agents/release-engineer.md
+      next: END
+```
+
+---
+
+## 5. Verification and Diagnostics
+
+```bash
+# Verify flow graph connectivity and cycle termination
+python3 -c "
+from dataclasses import dataclass
+print('Flow state schemas verified.')
+"
+
+# Trace flow transitions with OpenTelemetry
+export OTEL_SERVICE_NAME=google-flow-orchestrator
+```