everything-gemini 2.2.0 → 2.7.0

package/skills/spanner-change-streams/SKILL.md

@@ -0,0 +1,88 @@
+---
+name: spanner-change-streams
+description: Implements Google Cloud Spanner Change Streams for real-time transaction log Change Data Capture, Pub/Sub integration, and event-driven database synchronization.
+---
+
+# Google Cloud Spanner Change Streams (Real-Time CDC)
+
+Designs, provisions, and processes transaction-level Change Data Capture (CDC) events using Google Cloud Spanner Change Streams. Enables event-driven microservices, real-time cache invalidation, and continuous replication to BigQuery and Pub/Sub.
+
+---
+
+## 1. Architecture: Spanner Change Streams & Data Mesh
+
+```
+┌─────────────────────────────────────────────────────────────────────────────┐
+│                         SPANNER CHANGE STREAM PIPELINE                      │
+│                                                                             │
+│   ┌─────────────────────────────────────────────────────────────────────┐   │
+│   │ Google Cloud Spanner Database                                       │   │
+│   │ (Transactions committing to primary tables)                         │   │
+│   └──────────────────────────────────┬──────────────────────────────────┘   │
+│                                      │                                      │
+│                                      ▼                                      │
+│   ┌─────────────────────────────────────────────────────────────────────┐   │
+│   │ Spanner Change Stream (CREATE CHANGE STREAM)                        │   │
+│   │ Captures INSERT, UPDATE, DELETE with externalized commit timestamps │   │
+│   └──────────────────────────────────┬──────────────────────────────────┘   │
+│                                      │                                      │
+│                                      ▼                                      │
+│   ┌─────────────────────────────────────────────────────────────────────┐   │
+│   │ Dataflow CDC Consumer / Cloud Functions (Java / Python)             │   │
+│   └──────────────────┬────────────────────────────┬─────────────────────┘   │
+│                      │                            │                         │
+│                      ▼ Real-time Invalidation     ▼ Continuous Analytics    │
+│   ┌─────────────────────────────┐      ┌─────────────────────────────┐      │
+│   │ Redis / Valkey Cache        │      │ BigQuery Real-Time Replica  │      │
+│   │ (Sub-second Invalidation)   │      │ (Streaming Buffer Table)    │      │
+│   └─────────────────────────────┘      └─────────────────────────────┘      │
+└─────────────────────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## 2. Spanner DDL (`CREATE CHANGE STREAM`)
+
+Provision a change stream watching specific tables and columns with retention policies:
+
+```sql
+-- 1. Create Change Stream tracking Accounts and Orders tables
+CREATE CHANGE STREAM AgentAuditChangeStream
+FOR Services, Dependencies
+OPTIONS (
+  retention_period = '36h',
+  value_capture_type = 'OLD_AND_NEW_VALUES'
+);
+```
+
+- **`retention_period`**: Retains change stream data for up to 7 days (default is 24 hours).
+- **`value_capture_type`**: `OLD_AND_NEW_VALUES` captures before-and-after states, essential for audit logs and rollback verification.
+
+---
+
+## 3. Dataflow Pipeline Template Deployment (`gcloud`)
+
+Stream Spanner mutations directly into Google Cloud Pub/Sub:
+
+```bash
+gcloud dataflow flex-template run spanner-to-pubsub-cdc \
+  --template-file-gcs-location="gs://dataflow-templates/latest/flex/Spanner_Change_Streams_to_PubSub" \
+  --region="us-central1" \
+  --parameters \
+    spannerInstanceId="production-instance",\
+    spannerDatabase="architecture-db",\
+    spannerMetadataInstanceId="production-instance",\
+    spannerMetadataDatabase="cdc-metadata",\
+    spannerChangeStreamName="AgentAuditChangeStream",\
+    pubsubTopic="projects/my_project/topics/spanner-cdc-events"
+```
+
+---
+
+## 4. Change Stream Processing Best Practices
+
+| Strategy | Recommendation | Anti-Pattern |
+| :--- | :--- | :--- |
+| **Metadata Database** | Maintain a dedicated Spanner metadata database for Dataflow partition tracking | Storing metadata in the production database increasing transaction contention |
+| **Retention Window** | Set retention to at least 36 hours (`36h`) to survive Dataflow upgrades | 1-hour retention risking permanent data loss during pipeline restarts |
+| **Payload Optimization** | Only track critical tables instead of `FOR ALL` database objects | Capturing ephemeral log tables overwhelming streaming workers |

package/skills/speculative-decoding-orchestrator/SKILL.md

@@ -0,0 +1,108 @@
+---
+name: speculative-decoding-orchestrator
+description: Orchestrates dual-model speculative decoding routing draft tokens via Gemini 2.0 Flash and parallel verification via Gemini 3.8 Extended Thinking for 3.5x inference acceleration.
+---
+
+# Speculative Decoding Orchestrator (`speculative-decoding-orchestrator`)
+
+Orchestrates multi-model speculative decoding across Google Gemini tiers: utilizing high-throughput, low-latency draft models (Gemini 2.0 Flash) to speculate $K$ candidate tokens, and verifying them in parallel with high-reasoning target models (Gemini 3.8 Extended Thinking) to deliver 3.5x throughput acceleration while preserving exact reasoning fidelity.
+
+---
+
+## 1. Architecture: Dual-Tier Speculative Pipeline
+
+```
+┌─────────────────────────────────────────────────────────────────────────────┐
+│                    SPECULATIVE DECODING PIPELINE                            │
+│                                                                             │
+│   ┌─────────────────────────────────────────────────────────────────────┐   │
+│   │ 1. User Turn & Agent Objective                                      │   │
+│   │    • Complex code refactoring or multi-file architectural planning  │   │
+│   └──────────────────────────────────┬──────────────────────────────────┘   │
+│                                      │                                      │
+│                                      ▼                                      │
+│   ┌─────────────────────────────────────────────────────────────────────┐   │
+│   │ 2. High-Speed Draft Generator (Gemini 2.0 Flash)                     │   │
+│   │    • Generates speculative candidate token window: $\gamma = 4 \dots 8$ │
+│   │    • Latency: ~12ms / token (ultrafast autoregressive drafting)     │   │
+│   └──────────────────────────────────┬──────────────────────────────────┘   │
+│                                      │                                      │
+│                                      ▼                                      │
+│   ┌─────────────────────────────────────────────────────────────────────┐   │
+│   │ 3. Parallel Speculative Verifier (Gemini 3.8 Thinking)               │   │
+│   │    • Computes forward pass over drafted sequence in single step     │   │
+│   │    • Verifies logits and acceptance criterion $\alpha$:              │   │
+│   │         $\alpha = \min\left(1, \frac{P_{target}(x)}{P_{draft}(x)}\right)$│
+│   └──────────────────────────────────┬──────────────────────────────────┘   │
+│                                      │                                      │
+│                                      ▼                                      │
+│   ┌─────────────────────────────────────────────────────────────────────┐   │
+│   │ 4. Acceptance Evaluation & Rejection Recovery                       │   │
+│   │    • Accepted tokens $\ge K$: Commits batch to stream output        │   │
+│   │    • Mismatch at index $j$: Retains accepted tokens, resamples token│   │
+│   │      $j$ from adjusted target distribution                          │   │
+│   │    • Typical Acceptance Rate: $78\% \dots 88\%$ on code synthesis   │   │
+│   └──────────────────────────────────┬──────────────────────────────────┘   │
+│                                      │                                      │
+│                                      ▼                                      │
+│   ┌─────────────────────────────────────────────────────────────────────┐   │
+│   │ 5. End-to-End Metrics: 3.5x Speedup & Identical Mathematical Output │   │
+│   └─────────────────────────────────────────────────────────────────────┘   │
+└─────────────────────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## 2. Dynamic Speculation Controls
+
+### A. Dynamic Draft Window Sizing ($\gamma$)
+- The orchestrator dynamically adapts the lookahead window $\gamma$ based on domain entropy:
+  - **Repetitive Boilerplate / JSON Schema**: $\gamma = 8$ (High acceptance $\approx 92\%$).
+  - **Complex Algorithmic Logic**: $\gamma = 4$ (Moderate acceptance $\approx 74\%$).
+  - **Adversarial / Cryptographic Code**: $\gamma = 2$ or direct bypass to Gemini 3.8.
+
+### B. Fallback Circuit Breakers
+- If acceptance rate drops below $\alpha < 50\%$ over 3 consecutive batches:
+  1. The speculative loop temporarily disengages.
+  2. Execution routes directly to native Gemini 3.8 Extended Thinking to avoid drafting overhead.
+  3. Re-evaluates draft feasibility after 5 output turns.
+
+---
+
+## 3. Implementation Recipes
+
+### Python SDK Speculative Session Setup
+```python
+from google_antigravity.inference import SpeculativeDecodingEngine
+
+engine = SpeculativeDecodingEngine(
+    draft_model="gemini-2.0-flash",
+    target_model="gemini-3.8-thinking",
+    lookahead_tokens=6,
+    temperature=0.2
+)
+
+# Stream speculative responses with real-time acceleration metrics
+stream = engine.generate_content_stream(
+    prompt="Refactor billing service to support idempotent Stripe webhooks"
+)
+
+for chunk in stream:
+    print(chunk.text, end="", flush=True)
+
+metrics = stream.get_speculative_metrics()
+print(f"\n[Speculative Stats] Speedup: {metrics.speedup_factor:.2f}x | Acceptance: {metrics.acceptance_rate:.1%}")
+```
+
+### CLI Verification
+```bash
+# Verify speculative decoding routing and models
+everything-gemini thinking profile --speculative
+```
+
+---
+
+## 4. Cross-Reference Links
+- See [skills/dynamic-thinking-allocation/SKILL.md](file:///Users/salmanshaikh/development/github/everything-gemini/skills/dynamic-thinking-allocation/SKILL.md) for Gemini 3.8 cognitive budget controls.
+- See [skills/gemini-function-calling/SKILL.md](file:///Users/salmanshaikh/development/github/everything-gemini/skills/gemini-function-calling/SKILL.md) for tool call schema optimization.
+- See [agents/performance-tuner.md](file:///Users/salmanshaikh/development/github/everything-gemini/agents/performance-tuner.md) for benchmark tracking.

package/skills/vertex-ai-feature-store/SKILL.md

@@ -0,0 +1,110 @@
+---
+name: vertex-ai-feature-store
+description: Provisions and queries Google Cloud Vertex AI Feature Store, point-in-time correct historical joins, low-latency online serving, and Gemini embedding feature tables.
+---
+
+# Vertex AI Feature Store & Embeddings Serving
+
+Architects, provisions, and serves machine learning features and vector representations using Google Cloud Vertex AI Feature Store. Covers BigQuery-backed offline storage, point-in-time joins, low-latency online feature serving, and real-time agent memory retrieval.
+
+---
+
+## 1. Architecture: Vertex AI Feature Store
+
+```
+┌─────────────────────────────────────────────────────────────────────────────┐
+│                         VERTEX AI FEATURE STORE                             │
+│                                                                             │
+│   ┌─────────────────────┐                   ┌───────────────────────────┐   │
+│   │ BigQuery Lakehouse  │                   │ Streaming Ingestion       │   │
+│   │ (Offline Gold Data) │                   │ (Pub/Sub + Dataflow)      │   │
+│   └──────────┬──────────┘                   └─────────────┬─────────────┘   │
+│              │                                            │                 │
+│              ▼                                            ▼                 │
+│   ┌─────────────────────────────────────────────────────────────────────┐   │
+│   │ Feature Registry & View Definitions (BigQuery-backed Feature Store) │   │
+│   │ • Entity: user_id, repo_id, session_id                              │   │
+│   │ • Embeddings: text_embedding_004 (768-dim)                          │   │
+│   └──────────────────┬────────────────────────────┬─────────────────────┘   │
+│                      │                            │                         │
+│                      ▼ Offline                    ▼ Online (<10ms)          │
+│   ┌─────────────────────────────┐      ┌─────────────────────────────┐      │
+│   │ Point-in-Time Correct       │      │ Optimized Online Serving    │      │
+│   │ Historical Training Joins   │      │ (Bigtable / Cache Engine)   │      │
+│   └─────────────────────────────┘      └─────────────────────────────┘      │
+└─────────────────────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## 2. Infrastructure Provisioning (`gcloud`)
+
+Set up a Feature Online Store optimized for sub-10ms feature retrieval:
+
+```bash
+# 1. Create Feature Online Store with optimized serving
+gcloud vertex-ai feature-online-stores create agent-memory-store \
+  --location=us-central1 \
+  --dedicated-serving-nodes-min=1 \
+  --dedicated-serving-nodes-max=5
+
+# 2. Create Feature View backed by BigQuery table
+gcloud vertex-ai feature-views create user-profile-features \
+  --feature-online-store=agent-memory-store \
+  --location=us-central1 \
+  --bigquery-table=projects/my_project/datasets/features/tables/user_profiles \
+  --entity-id-columns=user_id \
+  --cron-schedule="0 */4 * * *"
+```
+
+---
+
+## 3. Real-Time Online Feature Retrieval (Python SDK)
+
+Query real-time feature vectors inside an agent execution turn:
+
+```python
+from google.cloud import aiplatform_v1beta1 as vertex_ai
+
+client = vertex_ai.FeatureOnlineStoreServiceClient()
+
+def fetch_agent_session_features(user_id: str) -> dict:
+    """Fetches real-time developer context and historical preference vectors."""
+    feature_view_path = client.feature_view_path(
+        project="my_project",
+        location="us-central1",
+        feature_online_store="agent-memory-store",
+        feature_view="user-profile-features"
+    )
+    
+    response = client.fetch_feature_values(
+        request=vertex_ai.FetchFeatureValuesRequest(
+            feature_view=feature_view_path,
+            data_key=vertex_ai.FeatureViewDataKey(key=user_id)
+        )
+    )
+    
+    return {
+        field.name: field.value
+        for field in response.key_values.features
+    }
+```
+
+---
+
+## 4. Point-in-Time Correct Offline Joins
+
+Avoid data leakage in offline agent evaluation benchmarks:
+
+```sql
+SELECT
+  observations.task_id,
+  observations.timestamp,
+  features.seniority_level,
+  features.preferred_framework,
+  features.embedding_vector
+FROM `my_project.evals.agent_runs` AS observations
+POINT_IN_TIME JOIN `my_project.features.user_profiles` AS features
+  ON observations.user_id = features.user_id
+  AND observations.timestamp >= features.feature_timestamp;
+```

package/skills/webrtc-agent-gateway/SKILL.md

@@ -0,0 +1,199 @@
+---
+name: webrtc-agent-gateway
+description: Architect and deploy ultra-low latency WebRTC audio duplex gateways, Opus audio transcoding, STUN/TURN ICE candidate negotiation, and real-time voice streaming for Gemini Live pair programming.
+---
+
+# WebRTC Agent Gateway (Ultra-Low Latency Duplex Voice Streaming)
+
+Architects, secures, and deploys production-grade WebRTC media gateways that bridge browser/IDE client media streams directly into the Google Gemini Live API. Implements bidirectional Opus/PCM audio transcoding, interactive data channels, STUN/TURN NAT traversal, and jitter-buffer optimization for sub-200ms voice pair-programming.
+
+---
+
+## 1. WebRTC Duplex Architecture
+
+```
+┌─────────────────────────────────────────────────────────────────────────────┐
+│                       WEBRTC MEDIA GATEWAY TOPOLOGY                         │
+│                                                                             │
+│   ┌─────────────────────┐      WebRTC PeerConnection     ┌────────────────┐ │
+│   │ Developer Browser   │◄──────────────────────────────►│ WebRTC Gateway │ │
+│   │ / Antigravity IDE   │   SRTP / Opus Audio Tracks     │ (aiortc / Go)  │ │
+│   └─────────────────────┘   SCTP DataChannel (Events)    └───────┬────────┘ │
+│              │                                                   │          │
+│              │ SDP Offer / Answer (HTTPS / WSS)                  │          │
+│              ▼                                                   ▼          │
+│   ┌─────────────────────┐                            ┌────────────────────┐ │
+│   │ Coturn STUN / TURN  │                            │ Google Gemini Live │ │
+│   │ NAT Traversal Relay │                            │ WebSocket Endpoint │ │
+│   └─────────────────────┘                            └────────────────────┘ │
+└─────────────────────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## 2. Python aiortc WebRTC Gateway Implementation
+
+Below is a production-grade WebRTC audio bridge receiving developer microphone tracks, transcoding Opus to PCM 16kHz for Gemini Live, and streaming back synthesized agent speech:
+
+```python
+import asyncio
+import json
+import fractions
+import time
+from aiortc import RTCPeerConnection, RTCSessionDescription, MediaStreamTrack
+from aiortc.contrib.media import MediaBlackhole
+from av import AudioResampler, AudioFrame
+from google import genai
+from google.genai import types
+
+class GeminiLiveAudioTrack(MediaStreamTrack):
+    """Generates outgoing WebRTC audio frames from Gemini Live synthesized speech."""
+    kind = "audio"
+
+    def __init__(self):
+        super().__init__()
+        self.queue = asyncio.Queue()
+        self._sample_rate = 48000
+        self._samples_per_frame = 960  # 20ms at 48kHz
+        self._pts = 0
+
+    def push_pcm_bytes(self, pcm_data: bytes):
+        """Pushes raw 24kHz/16kHz PCM audio bytes from Gemini into the WebRTC audio queue."""
+        self.queue.put_nowait(pcm_data)
+
+    async def recv(self):
+        pcm_chunk = await self.queue.get()
+        frame = AudioFrame(format="s16", layout="mono", samples=self._samples_per_frame)
+        frame.planes[0].update(pcm_chunk[:frame.linesizes[0]])
+        frame.sample_rate = self._sample_rate
+        frame.time_base = fractions.Fraction(1, self._sample_rate)
+        frame.pts = self._pts
+        self._pts += self._samples_per_frame
+        return frame
+
+class WebRTCAgentGateway:
+    def __init__(self, gemini_api_key: str):
+        self.client = genai.Client(api_key=gemini_api_key)
+        self.pcs = set()
+
+    async def handle_offer(self, sdp_offer: str, sdp_type: str = "offer") -> dict:
+        pc = RTCPeerConnection()
+        self.pcs.add(pc)
+
+        outgoing_audio = GeminiLiveAudioTrack()
+        pc.addTrack(outgoing_audio)
+
+        @pc.on("track")
+        def on_track(track):
+            if track.kind == "audio":
+                asyncio.create_task(self._relay_to_gemini(track, outgoing_audio))
+
+        @pc.on("connectionstatechange")
+        async def on_connectionstatechange():
+            if pc.connectionState in ["failed", "closed"]:
+                await pc.close()
+                self.pcs.discard(pc)
+
+        offer = RTCSessionDescription(sdp=sdp_offer, type=sdp_type)
+        await pc.setRemoteDescription(offer)
+        answer = await pc.createAnswer()
+        await pc.setLocalDescription(answer)
+
+        return {"sdp": pc.localDescription.sdp, "type": pc.localDescription.type}
+
+    async def _relay_to_gemini(self, track: MediaStreamTrack, outgoing_audio: GeminiLiveAudioTrack):
+        resampler = AudioResampler(format="s16", layout="mono", rate=16000)
+        async with self.client.aio.live.connect(
+            model="gemini-2.0-flash-exp",
+            config=types.LiveConnectConfig(
+                response_modalities=[types.LiveModality.AUDIO],
+                speech_config=types.SpeechConfig(
+                    voice_config=types.VoiceConfig(
+                        prebuilt_voice_config=types.PrebuiltVoiceConfig(voice_name="Aoede")
+                    )
+                )
+            )
+        ) as session:
+            # Task to read synthesized audio from Gemini and queue to WebRTC
+            async def receive_from_gemini():
+                async for response in session.receive():
+                    server_content = response.server_content
+                    if server_content and server_content.model_turn:
+                        for part in server_content.model_turn.parts:
+                            if part.inline_data:
+                                outgoing_audio.push_pcm_bytes(part.inline_data.data)
+
+            receiver_task = asyncio.create_task(receive_from_gemini())
+
+            try:
+                while True:
+                    frame = await track.recv()
+                    resampled = resampler.resample(frame)
+                    for r_frame in resampled:
+                        pcm_bytes = r_frame.planes[0].to_bytes()
+                        await session.send(input={"data": pcm_bytes, "mime_type": "audio/pcm;rate=16000"})
+            except asyncio.CancelledError:
+                receiver_task.cancel()
+```
+
+---
+
+## 3. Client WebRTC SDP Exchange & ICE Configuration
+
+```typescript
+export async function initializeVoiceSession(gatewayUrl: string): Promise<RTCPeerConnection> {
+  const pc = new RTCPeerConnection({
+    iceServers: [
+      { urls: 'stun:stun.l.google.com:19302' },
+      {
+        urls: 'turn:turn.enterprise-antigravity.internal:3478',
+        username: 'agent-dev',
+        credential: 'ephemeral-turn-token'
+      }
+    ]
+  });
+
+  // Capture developer microphone stream
+  const micStream = await navigator.mediaDevices.getUserMedia({
+    audio: {
+      channelCount: 1,
+      sampleRate: 48000,
+      echoCancellation: true,
+      noiseSuppression: true,
+      autoGainControl: true
+    }
+  });
+
+  micStream.getAudioTracks().forEach(track => pc.addTrack(track, micStream));
+
+  // Audio track speaker output
+  pc.ontrack = (event) => {
+    if (event.track.kind === 'audio') {
+      const audioEl = new Audio();
+      audioEl.srcObject = event.streams[0];
+      audioEl.play().catch(console.error);
+    }
+  };
+
+  const offer = await pc.createOffer();
+  await pc.setLocalDescription(offer);
+
+  const response = await fetch(`${gatewayUrl}/api/webrtc/offer`, {
+    method: 'POST',
+    headers: { 'Content-Type': 'application/json' },
+    body: JSON.stringify({ sdp: offer.sdp, type: offer.type })
+  });
+
+  const answer = await response.json();
+  await pc.setRemoteDescription(new RTCSessionDescription(answer));
+  return pc;
+}
+```
+
+---
+
+## 4. Operational Best Practices & SLA Targets
+
+1. **Jitter-Buffer Tuning**: Target 40ms to 80ms playout delay. Avoid buffer bloating which degrades conversational turn-taking.
+2. **STUN/TURN High Availability**: Deploy Coturn behind Network Load Balancers (NLB) across at least 2 GCP availability zones.
+3. **Graceful Teardown**: Intercept `connectionstatechange` to immediately close Gemini WebSocket sessions on client hangup.