everything-gemini 2.2.0 → 2.7.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (51) hide show
  1. package/BENCHMARK.md +33 -33
  2. package/CHANGELOG.md +84 -0
  3. package/README.md +91 -6
  4. package/agents/cross-repo-orchestrator.md +33 -0
  5. package/agents/darwin-prompt-engineer.md +35 -0
  6. package/agents/resilience-engineer.md +36 -0
  7. package/agents/swarm-coordinator.md +35 -0
  8. package/bin/everything-gemini +1175 -13
  9. package/docs/LAUNCH_KIT_PHP2SHELL.md +240 -0
  10. package/docs/LAUNCH_KIT_v2.4.0.md +164 -0
  11. package/docs/LAUNCH_KIT_v2.5.0.md +187 -0
  12. package/docs/LAUNCH_KIT_v2.6.0.md +174 -0
  13. package/docs/index.html +32 -14
  14. package/examples/python-sdk/cross_repo_swarm_and_compaction.py +175 -0
  15. package/examples/python-sdk/ephemeral_swarm_orchestration.py +169 -0
  16. package/examples/python-sdk/resilience_and_telemetry_pipeline.py +183 -0
  17. package/hooks/context-compactor/hooks.json +16 -0
  18. package/hooks/context-compactor/scripts/compact-context.sh +40 -0
  19. package/hooks/drift-detector/hooks.json +16 -0
  20. package/hooks/drift-detector/scripts/detect-drift.sh +50 -0
  21. package/hooks/thinking-budget/hooks.json +16 -0
  22. package/hooks/thinking-budget/scripts/budget-thinking.sh +56 -0
  23. package/package.json +1 -1
  24. package/skills/adversarial-fuzz-generator/SKILL.md +95 -0
  25. package/skills/context-distillation-compressor/SKILL.md +109 -0
  26. package/skills/cost-oracle-finops/SKILL.md +142 -0
  27. package/skills/cross-repo-swarm-sync/SKILL.md +122 -0
  28. package/skills/cve-remediation-orchestrator/SKILL.md +111 -0
  29. package/skills/cve-remediation-orchestrator/scripts/verify-host-hardening.sh +72 -0
  30. package/skills/dynamic-thinking-allocation/SKILL.md +105 -0
  31. package/skills/ebpf-microvm-sandbox/SKILL.md +113 -0
  32. package/skills/ebpf-sandbox-enforcer/SKILL.md +101 -0
  33. package/skills/envoy-ai-gateway/SKILL.md +175 -0
  34. package/skills/ephemeral-swarm-orchestrator/SKILL.md +124 -0
  35. package/skills/gemini-live-multimodal/SKILL.md +149 -0
  36. package/skills/gemini-live-screen-vision/SKILL.md +109 -0
  37. package/skills/genetic-prompt-cross-pollinator/SKILL.md +110 -0
  38. package/skills/harness3-mutation-testing/SKILL.md +120 -0
  39. package/skills/instinct-distillation-compiler/SKILL.md +103 -0
  40. package/skills/opentelemetry-agent-observability/SKILL.md +139 -0
  41. package/skills/php2shell-version-auditor/SKILL.md +95 -0
  42. package/skills/phpcheck-server/SKILL.md +102 -0
  43. package/skills/phpcheck-server/scripts/audit-server.sh +112 -0
  44. package/skills/phpcheck-url/SKILL.md +72 -0
  45. package/skills/phpcheck-url/scripts/phpcheck.js +160 -0
  46. package/skills/quantum-safe-cryptography/SKILL.md +84 -0
  47. package/skills/spanner-change-streams/SKILL.md +88 -0
  48. package/skills/speculative-decoding-orchestrator/SKILL.md +108 -0
  49. package/skills/vertex-ai-feature-store/SKILL.md +110 -0
  50. package/skills/webrtc-agent-gateway/SKILL.md +199 -0
  51. package/tests/cli.test.js +253 -14
@@ -0,0 +1,113 @@
1
+ ---
2
+ name: ebpf-microvm-sandbox
3
+ description: Enforces rootless ephemeral Micro-VM isolation, Linux Landlock filesystem jails, and Cilium eBPF network filtering for zero-trust speculative tool and code execution.
4
+ ---
5
+
6
+ # eBPF Micro-VM Sandbox (`ebpf-microvm-sandbox`)
7
+
8
+ Provides kernel-enforced rootless ephemeral Micro-VM isolation (Firecracker and gVisor), Linux Landlock path jails, seccomp-bpf syscall filtering, and Cilium eBPF network security for untrusted, speculative tool execution in Google Antigravity and Gemini agent loops.
9
+
10
+ ---
11
+
12
+ ## 1. Architectural Model: Kernel-Enforced Isolation Hierarchy
13
+
14
+ ```
15
+ ┌─────────────────────────────────────────────────────────────────────────────┐
16
+ │ ZERO-TRUST AGENT SANDBOX RUNTIME │
17
+ │ │
18
+ │ ┌─────────────────────────────────────────────────────────────────────┐ │
19
+ │ │ 1. Antigravity Agent Runtime (Host Process) │ │
20
+ │ │ • Speculative tool proposal (e.g. `run_command`, `eval_python`) │ │
21
+ │ │ • Intercepted by PreToolUse Lifecycle Security Hook │ │
22
+ │ └──────────────────────────────────┬──────────────────────────────────┘ │
23
+ │ │ │
24
+ │ ▼ │
25
+ │ ┌─────────────────────────────────────────────────────────────────────┐ │
26
+ │ │ 2. OPA Policy Gatekeeper & Sandbox Dispatcher │ │
27
+ │ │ • Validates command safety signature and environment constraints │ │
28
+ │ │ • Allocates ephemeral rootless micro-VM workspace │ │
29
+ │ └──────────────────────────────────┬──────────────────────────────────┘ │
30
+ │ │ │
31
+ │ ▼ │
32
+ │ ┌─────────────────────────────────────────────────────────────────────┐ │
33
+ │ │ 3. Ephemeral Micro-VM Jail (Firecracker / gVisor runsc) │ │
34
+ │ │ ┌────────────────────────────────────────────────────────────┐ │ │
35
+ │ │ │ A. Filesystem: Linux Landlock V4 ephemeral copy-on-write │ │ │
36
+ │ │ │ • Host files outside sandbox mounted READ-ONLY │ │ │
37
+ │ │ │ • Temp workdir deleted on tool completion │ │ │
38
+ │ │ ├────────────────────────────────────────────────────────────┤ │ │
39
+ │ │ │ B. Syscalls: Strict Seccomp-BPF whitelist filter │ │ │
40
+ │ │ │ • Blocks ptrace, mount, bpf, reboot, setuid, unshare │ │ │
41
+ │ │ ├────────────────────────────────────────────────────────────┤ │ │
42
+ │ │ │ C. Network: Cilium eBPF Socket Filtering & Egress Policy │ │ │
43
+ │ │ │ • Strict CIDR whitelist (e.g. npmjs, pypi, googleapis) │ │ │
44
+ │ │ │ • Blocks RFC 1918 internal subnets & cloud metadata IPs │ │ │
45
+ │ │ └────────────────────────────────────────────────────────────┘ │ │
46
+ │ └──────────────────────────────────┬──────────────────────────────────┘ │
47
+ │ │ │
48
+ │ ▼ │
49
+ │ ┌─────────────────────────────────────────────────────────────────────┐ │
50
+ │ │ 4. Execution Telemetry & Differential Sync │ │
51
+ │ │ • Captured stdout, stderr, exit code, and modified file diffs │ │
52
+ │ │ • Automatic rollback on policy violation or timeout abort │ │
53
+ │ └─────────────────────────────────────────────────────────────────────┘ │
54
+ └─────────────────────────────────────────────────────────────────────────────┘
55
+ ```
56
+
57
+ ---
58
+
59
+ ## 2. Kernel Security Primitives
60
+
61
+ ### A. Linux Landlock V4 Path Rules
62
+ - Speculative tools receive scoped path access:
63
+ - `LANDLOCK_ACCESS_FS_READ`: Allowed on workspace root.
64
+ - `LANDLOCK_ACCESS_FS_WRITE`: Confined strictly to `/tmp/agent-sandbox/<execution-id>`.
65
+ - Sensitive paths (`~/.ssh`, `~/.aws`, `~/.config/gcloud`, `/etc/shadow`) are masked with empty dummy inodes.
66
+
67
+ ### B. Seccomp-BPF Syscall Restriction
68
+ - The sandbox enforces an allow-list BPF program at `execve`:
69
+ ```text
70
+ ALLOWED: read, write, openat, close, fstat, lseek, mmap, mprotect, munmap, brk, exit_group
71
+ DENIED (SIGKILL): ptrace, process_vm_readv, init_module, kexec_load, bpf, clone(NEWUSER)
72
+ ```
73
+
74
+ ### C. Cilium eBPF Network Quarantine
75
+ - Egress is strictly blocked to Cloud Metadata Endpoints:
76
+ - `169.254.169.254/32` (AWS/GCP Instance Metadata Service)
77
+ - `10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16` (Internal LAN)
78
+ - Only TLS port 443 egress to explicit DNS-validated dependency endpoints is permitted.
79
+
80
+ ---
81
+
82
+ ## 3. Operational Recipes
83
+
84
+ ### Launching an Isolated Speculative Command
85
+ ```bash
86
+ # Verify sandbox status and host kernel security primitives
87
+ everything-gemini doctor --profile full
88
+
89
+ # Run speculative untrusted build in micro-VM jail
90
+ everything-gemini resilience inject --target "speculative-build" --sandbox microvm
91
+ ```
92
+
93
+ ### Landlock Python Runner Pattern
94
+ ```python
95
+ from google_antigravity.security import EphemeralMicroVMSandbox
96
+
97
+ with EphemeralMicroVMSandbox(
98
+ memory_mb=512,
99
+ cpu_limit=1.0,
100
+ allow_network=False,
101
+ timeout_sec=30
102
+ ) as sandbox:
103
+ result = sandbox.exec(["python3", "-c", "import os; print(os.listdir('.'))"])
104
+ print(f"Sandbox Exit Code: {result.exit_code}")
105
+ print(f"Sandbox Output: {result.stdout}")
106
+ ```
107
+
108
+ ---
109
+
110
+ ## 4. Cross-Reference & Links
111
+ - See [skills/ebpf-sandbox-enforcer/SKILL.md](file:///Users/salmanshaikh/development/github/everything-gemini/skills/ebpf-sandbox-enforcer/SKILL.md) for Landlock baseline rules.
112
+ - See [skills/opa-gatekeeper-policy/SKILL.md](file:///Users/salmanshaikh/development/github/everything-gemini/skills/opa-gatekeeper-policy/SKILL.md) for OPA policy syntax.
113
+ - See [agents/security-auditor.md](file:///Users/salmanshaikh/development/github/everything-gemini/agents/security-auditor.md) for automated vulnerability auditing.
@@ -0,0 +1,101 @@
1
+ ---
2
+ name: ebpf-sandbox-enforcer
3
+ description: Enforces zero-trust rootless micro-VM and eBPF kernel security boundaries for speculative agent tool executions and container sandboxes.
4
+ ---
5
+
6
+ # eBPF Sandbox Enforcer & Zero-Trust Tool Isolation
7
+
8
+ Implements kernel-level zero-trust security isolation, rootless micro-VM sandboxing, Landlock filesystem restriction, and Cilium eBPF network filtering for speculative agent tool executions within Google Antigravity and Everything Gemini workflows.
9
+
10
+ ---
11
+
12
+ ## 1. Architecture: Kernel-Level Agent Sandboxing
13
+
14
+ ```
15
+ ┌─────────────────────────────────────────────────────────────────────────────┐
16
+ │ KERNEL-LEVEL AGENT SECURITY BOUNDARY │
17
+ │ │
18
+ │ ┌─────────────────────────────────────────────────────────────────────┐ │
19
+ │ │ Speculative Agent Tool Execution (`run_command`, `write_to_file`) │ │
20
+ │ └──────────────────────────────────┬──────────────────────────────────┘ │
21
+ │ │ │
22
+ │ ▼ │
23
+ │ ┌─────────────────────────────────────────────────────────────────────┐ │
24
+ │ │ 1. User-Space Policy Interceptor (`gemini-shield` & OPA Rego) │ │
25
+ │ │ • Validates command AST, blocks `rm -rf`, detects token leakages │ │
26
+ │ └──────────────────────────────────┬──────────────────────────────────┘ │
27
+ │ │ │
28
+ │ ▼ │
29
+ │ ┌─────────────────────────────────────────────────────────────────────┐ │
30
+ │ │ 2. Linux Landlock & Seccomp-BPF Syscall Filter │ │
31
+ │ │ • Traps unauthorized `execve`, `ptrace`, `setuid`, and `chroot` │ │
32
+ │ │ • Restricts filesystem I/O strictly to isolated `./scratch/` dir │ │
33
+ │ └──────────────────────────────────┬──────────────────────────────────┘ │
34
+ │ │ │
35
+ │ ▼ │
36
+ │ ┌─────────────────────────────────────────────────────────────────────┐ │
37
+ │ │ 3. Cilium eBPF Socket Filter (L3/L4/L7 Network Egress) │ │
38
+ │ │ • Blocks outbound internet access to non-whitelisted IPs │ │
39
+ │ │ • Allows Google Cloud APIs (`*.googleapis.com`) on TLS 443 │ │
40
+ │ │ • Drops all lateral movement probes to internal Kubernetes pods │ │
41
+ │ └─────────────────────────────────────────────────────────────────────┘ │
42
+ └─────────────────────────────────────────────────────────────────────────────┘
43
+ ```
44
+
45
+ ---
46
+
47
+ ## 2. Kernel Enforcement Mechanisms
48
+
49
+ ### A. Landlock Filesystem Sandboxing
50
+ Speculative agent mutations are restricted to the local workspace directory or an ephemeral scratch area. System directories (`/etc`, `/usr`, `/root`, `~/.ssh`) are marked strictly `LANDLOCK_ACCESS_FS_READ` or completely denied:
51
+
52
+ ```c
53
+ // Example Landlock ruleset structure for agent tool sandbox
54
+ #include <linux/landlock.h>
55
+ #include <sys/prctl.h>
56
+
57
+ int enforce_agent_landlock(const char *allowed_dir) {
58
+ int ruleset_fd = landlock_create_ruleset(&ruleset_attr, sizeof(ruleset_attr), 0);
59
+ // Add rule: allowed_dir gets read/write/truncate/execute
60
+ // System directories remain read-only or inaccessible
61
+ prctl(PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0);
62
+ landlock_restrict_self(ruleset_fd, 0);
63
+ return 0;
64
+ }
65
+ ```
66
+
67
+ ### B. Cilium eBPF Network Security Policy (CNP)
68
+ Prevents data exfiltration or reverse shells when the agent runs third-party build scripts or untrusted dependencies:
69
+
70
+ ```yaml
71
+ apiVersion: "cilium.io/v2"
72
+ kind: CiliumNetworkPolicy
73
+ metadata:
74
+ name: "agent-tool-sandbox-egress"
75
+ spec:
76
+ endpointSelector:
77
+ matchLabels:
78
+ role: "autonomous-agent-worker"
79
+ egress:
80
+ - toFQDNs:
81
+ - matchPattern: "*.googleapis.com"
82
+ - matchName: "github.com"
83
+ toPorts:
84
+ - ports:
85
+ - port: "443"
86
+ protocol: TCP
87
+ - toEntities:
88
+ - cluster
89
+ toPorts:
90
+ - ports:
91
+ - port: "53"
92
+ protocol: UDP
93
+ ```
94
+
95
+ ---
96
+
97
+ ## 3. Best Practices & Isolation Checklist
98
+
99
+ 1. **Rootless Execution**: Never execute autonomous agent commands as `root` (UID 0). Always run inside an unprivileged user namespace (`UID 10001`).
100
+ 2. **Ephemeral Disks**: Speculative test runs must target tmpfs or overlayfs volumes that can be unmounted and discarded in $<100\text{ms}$.
101
+ 3. **Audit Telemetry**: Feed eBPF syscall audit logs (`bpftool prog tracelog`) into the Mission Control UI to detect anomalous fork storms or socket scans.
@@ -0,0 +1,175 @@
1
+ ---
2
+ name: envoy-ai-gateway
3
+ description: Configure and deploy Envoy Proxy AI Gateway for Gemini models with token-aware rate limiting, multi-model circuit breakers, active health checks, and zero-trust mTLS termination.
4
+ ---
5
+
6
+ # Envoy Proxy AI Gateway (Multi-Model Traffic Director & Circuit Breaking)
7
+
8
+ Deploys and configures high-performance Envoy Proxy AI Gateways tailored for Google Gemini APIs and autonomous agent clusters. Provides token-aware rate limiting, active health checking, automatic circuit breaking across Gemini Flash, Pro, and Extended Thinking tiers, and zero-trust mTLS proxying.
9
+
10
+ ---
11
+
12
+ ## 1. AI Gateway Architecture
13
+
14
+ ```
15
+ ┌─────────────────────────────────────────────────────────────────────────────┐
16
+ │ ENVOY PROXY AI GATEWAY │
17
+ │ │
18
+ │ Agent Workflows (gRPC / HTTPS) │
19
+ │ │ │
20
+ │ ▼ │
21
+ │ ┌───────────────────────────┐ │
22
+ │ │ Envoy Proxy AI Gateway │◄──── Token-Aware Rate Limiter (Redis) │
23
+ │ │ (Filter Chain & Router) │◄──── W3C Tracecontext Injector │
24
+ │ └─────────────┬─────────────┘ │
25
+ │ │ │
26
+ │ ├── 90% Primary Route ──► Gemini 2.0 Flash (Fast Tool Use) │
27
+ │ │ │
28
+ │ ├── Circuit Broken? ──► Gemini 3.8 Pro (Fallback Tier) │
29
+ │ │ │
30
+ │ └── High Complexity? ──► Gemini 3.8 Thinking (Extended) │
31
+ └─────────────────────────────────────────────────────────────────────────────┘
32
+ ```
33
+
34
+ ---
35
+
36
+ ## 2. Envoy Proxy Configuration (`envoy.yaml`)
37
+
38
+ Below is the production Envoy proxy configuration routing requests to Google Gemini with outlier detection, retry backoff, and local rate limiting:
39
+
40
+ ```yaml
41
+ static_resources:
42
+ listeners:
43
+ - name: gemini_ingress_listener
44
+ address:
45
+ socket_address:
46
+ address: 0.0.0.0
47
+ port_value: 8443
48
+ filter_chains:
49
+ - transport_socket:
50
+ name: envoy.transport_sockets.tls
51
+ typed_config:
52
+ "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
53
+ common_tls_context:
54
+ tls_certificates:
55
+ - certificate_chain: { filename: "/etc/envoy/certs/server.crt" }
56
+ private_key: { filename: "/etc/envoy/certs/server.key" }
57
+ filters:
58
+ - name: envoy.filters.network.http_connection_manager
59
+ typed_config:
60
+ "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
61
+ stat_prefix: ingress_gemini
62
+ route_config:
63
+ name: gemini_api_routes
64
+ virtual_hosts:
65
+ - name: gemini_service
66
+ domains: ["*"]
67
+ routes:
68
+ - match:
69
+ prefix: "/v1beta/models/gemini-2.0-flash"
70
+ route:
71
+ cluster: gemini_api_upstream
72
+ timeout: 10s
73
+ retry_policy:
74
+ retry_on: "5xx,connect-failure,refused-stream,429"
75
+ num_retries: 3
76
+ retry_back_off:
77
+ base_interval: 250ms
78
+ max_interval: 2s
79
+ - match:
80
+ prefix: "/v1beta/models/gemini-3.8-thinking"
81
+ route:
82
+ cluster: gemini_api_upstream
83
+ timeout: 60s
84
+ http_filters:
85
+ - name: envoy.filters.http.local_ratelimit
86
+ typed_config:
87
+ "@type": type.googleapis.com/envoy.extensions.filters.http.local_ratelimit.v3.LocalRateLimit
88
+ stat_prefix: http_local_rate_limiter
89
+ token_bucket:
90
+ max_tokens: 1000
91
+ tokens_per_fill: 100
92
+ fill_interval: 1s
93
+ filter_enabled:
94
+ default_value: { numerator: 100, denominator: HUNDRED }
95
+ filter_enforced:
96
+ default_value: { numerator: 100, denominator: HUNDRED }
97
+ - name: envoy.filters.http.router
98
+ typed_config:
99
+ "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
100
+
101
+ clusters:
102
+ - name: gemini_api_upstream
103
+ type: LOGICAL_DNS
104
+ dns_lookup_family: V4_ONLY
105
+ connect_timeout: 2s
106
+ lb_policy: ROUND_ROBIN
107
+ circuit_breakers:
108
+ thresholds:
109
+ - priority: DEFAULT
110
+ max_connections: 500
111
+ max_pending_requests: 100
112
+ max_requests: 1000
113
+ max_retries: 3
114
+ outlier_detection:
115
+ consecutive_5xx: 3
116
+ interval: 10s
117
+ base_ejection_time: 30s
118
+ max_ejection_percent: 50
119
+ load_assignment:
120
+ cluster_name: gemini_api_upstream
121
+ endpoints:
122
+ - lb_endpoints:
123
+ - endpoint:
124
+ address:
125
+ socket_address:
126
+ address: generativelanguage.googleapis.com
127
+ port_value: 443
128
+ transport_socket:
129
+ name: envoy.transport_sockets.tls
130
+ typed_config:
131
+ "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
132
+ sni: generativelanguage.googleapis.com
133
+ ```
134
+
135
+ ---
136
+
137
+ ## 3. Kubernetes Deployment Manifest
138
+
139
+ ```yaml
140
+ apiVersion: apps/v1
141
+ kind: Deployment
142
+ metadata:
143
+ name: envoy-gemini-gateway
144
+ namespace: agent-runtime
145
+ spec:
146
+ replicas: 3
147
+ selector:
148
+ matchLabels:
149
+ app: envoy-gemini-gateway
150
+ template:
151
+ metadata:
152
+ labels:
153
+ app: envoy-gemini-gateway
154
+ spec:
155
+ containers:
156
+ - name: envoy
157
+ image: envoyproxy/envoy:v1.31-latest
158
+ ports:
159
+ - containerPort: 8443
160
+ volumeMounts:
161
+ - name: envoy-config
162
+ mountPath: /etc/envoy
163
+ volumes:
164
+ - name: envoy-config
165
+ configMap:
166
+ name: envoy-gemini-config
167
+ ```
168
+
169
+ ---
170
+
171
+ ## 4. Key Performance Indicators & Metrics
172
+
173
+ - **Ejection Rate (`envoy.cluster.outlier_detection.ejections_enforced`)**: Alerts on sustained API throttling.
174
+ - **Circuit Breakers (`envoy.cluster.circuit_breakers.default.rq_open`)**: Triggers immediate failover to backup model tiers.
175
+ - **P99 Proxy Overhead**: Sub-1.5ms added latency across TLS handshake and routing filters.
@@ -0,0 +1,124 @@
1
+ ---
2
+ name: ephemeral-swarm-orchestrator
3
+ description: Dynamically synthesizes, deploys, and reclaims ephemeral multi-agent swarms for cross-repository parallel refactoring and consensus aggregation.
4
+ ---
5
+
6
+ # Ephemeral Swarm Orchestrator (Swarm Intelligence)
7
+
8
+ Implements dynamic on-demand synthesis, parallel task decomposition, distributed cross-repository execution, consensus aggregation, and lifecycle reclamation of ephemeral micro-agent swarms for Google Gemini and Antigravity.
9
+
10
+ ---
11
+
12
+ ## 1. Architecture: Dynamic Ephemeral Micro-Agent Swarm
13
+
14
+ ```
15
+ ┌─────────────────────────────────────────────────────────────────────────────┐
16
+ │ EPHEMERAL SWARM ORCHESTRATION PIPELINE │
17
+ │ │
18
+ │ ┌─────────────────────────────────────────────────────────────────────┐ │
19
+ │ │ 1. Swarm Coordinator & Planner Decomposition │ │
20
+ │ │ • Complex multi-repo goal or massive refactor prompt │ │
21
+ │ │ • Decomposed into DAG of atomic micro-tasks with input contracts │ │
22
+ │ └──────────────────────────────────┬──────────────────────────────────┘ │
23
+ │ │ │
24
+ │ ▼ │
25
+ │ ┌─────────────────────────────────────────────────────────────────────┐ │
26
+ │ │ 2. Dynamic Ephemeral Specialist Synthesis │ │
27
+ │ │ • Synthesizes targeted micro-agent definitions in memory │ │
28
+ │ │ • E.g. `spanner-deadlock-tuner`, `grpc-schema-validator` │ │
29
+ │ │ • Mounts minimal progressive skills & scoped tool permissions │ │
30
+ │ └──────────────────────────────────┬──────────────────────────────────┘ │
31
+ │ │ │
32
+ │ ▼ │
33
+ │ ┌─────────────────────────────────────────────────────────────────────┐ │
34
+ │ │ 3. Parallel Worker Swarm Execution (Pub/Sub & Cloud Run Jobs) │ │
35
+ │ │ ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ │ │
36
+ │ │ │ Worker #1 (RepoA)│ │ Worker #2 (RepoB)│ │ Worker #3 (Infra)│ │ │
37
+ │ │ └────────┬─────────┘ └────────┬─────────┘ └────────┬─────────┘ │ │
38
+ │ └─────────────┼─────────────────────┼─────────────────────┼───────────┘ │
39
+ │ └─────────────────────┼─────────────────────┘ │
40
+ │ ▼ │
41
+ │ ┌─────────────────────────────────────────────────────────────────────┐ │
42
+ │ │ 4. Majority Consensus & Multi-Agent Voting │ │
43
+ │ │ • Aggregates diff proposals across parallel micro-agents │ │
44
+ │ │ • Conflict resolution via Byzantine/majority agreement matrix │ │
45
+ │ └──────────────────────────────────┬──────────────────────────────────┘ │
46
+ │ │ │
47
+ │ ▼ │
48
+ │ ┌─────────────────────────────────────────────────────────────────────┐ │
49
+ │ │ 5. Ephemeral Harness Teardown & Lifecycle Reclamation │ │
50
+ │ │ • Verifies property-based unit & integration tests │ │
51
+ │ │ • Distills learned instincts into persistent memory │ │
52
+ │ │ • Decommissions worker contexts and purges memory state │ │
53
+ │ └─────────────────────────────────────────────────────────────────────┘ │
54
+ └─────────────────────────────────────────────────────────────────────────────┘
55
+ ```
56
+
57
+ ---
58
+
59
+ ## 2. Core Operational Pillars
60
+
61
+ ### A. Dynamic Specialist Synthesis
62
+ Traditional multi-agent frameworks rely on rigid, static subagent registries. The Ephemeral Swarm Orchestrator dynamically constructs targeted agent personalities and tool contracts based on task requirements:
63
+
64
+ ```json
65
+ {
66
+ "swarm_id": "swarm-20261022-spanner-sync",
67
+ "coordinator": "swarm-coordinator",
68
+ "topology": "hierarchical",
69
+ "workers": [
70
+ {
71
+ "id": "worker-schema-evaluator",
72
+ "specialization": "spanner-ddl-auditor",
73
+ "scoped_skills": ["spanner-change-streams", "security-audit"],
74
+ "permitted_tools": ["view_file", "grep_search", "run_command"],
75
+ "token_budget": 8000
76
+ },
77
+ {
78
+ "id": "worker-migration-author",
79
+ "specialization": "liquibase-migration-writer",
80
+ "scoped_skills": ["database-migrator", "tdd-workflow"],
81
+ "permitted_tools": ["write_to_file", "replace_file_content", "run_command"],
82
+ "token_budget": 12000
83
+ }
84
+ ]
85
+ }
86
+ ```
87
+
88
+ ### B. Consensus & Voting Protocol
89
+ When multiple micro-agents generate speculative patches or architectural proposals, the swarm coordinator invokes a multi-agent voting round:
90
+ 1. **P2P Cross-Verification**: Worker 1 reviews Worker 2's AST patch.
91
+ 2. **Deterministic Test Execution**: Candidate diffs are executed against property-based tests in isolated ephemeral directories.
92
+ 3. **Quorum Verification**: A change requires $\ge 67\%$ consensus agreement before promotion into the primary branch.
93
+
94
+ ### C. Resource Reclamation & Zero Leaks
95
+ Upon completion or failure, the swarm executor executes cleanup routines:
96
+ * Terminates background worker processes and subprocess handles.
97
+ * Deletes scratch directories and temporary SQLite / DuckDB scratchpads.
98
+ * Commits learned operational invariants into `.agents/hooks/learned-instincts/`.
99
+
100
+ ---
101
+
102
+ ## 3. CLI Commands
103
+
104
+ ```bash
105
+ # 1. Spawn an ephemeral swarm with 3 parallel workers
106
+ everything-gemini swarm spawn "refactor auth middleware to zero-trust mTLS" --workers 3
107
+
108
+ # 2. Inspect active swarm topology and worker state
109
+ everything-gemini swarm status --json
110
+
111
+ # 3. Force teardown and reclaim all active worker resources
112
+ everything-gemini swarm reclaim --dry-run
113
+ ```
114
+
115
+ ---
116
+
117
+ ## 4. Anti-Patterns & Safety Guardrails
118
+
119
+ | Anti-Pattern | Operational Risk | Swarm Enforced Fix |
120
+ | :--- | :--- | :--- |
121
+ | **Unbounded Worker Spawning** | Exponential token burn and process fork bomb | Hard worker ceiling (`--max-workers 8`) enforced by `budget-guard`. |
122
+ | **Shared State Collisions** | Merge conflicts and dirty git tree states | Each worker operates in an isolated temporary worktree with git rollback snapshots. |
123
+ | **Zombie Subagents** | Orphaned background processes leaking memory | Mandatory timeout watchdog (`--timeout-sec 300`) with auto-reclamation. |
124
+ | **Single-Agent Bias** | Hallucinated code passes without review | Two-agent peer verification required on all synthesized diffs. |
@@ -0,0 +1,149 @@
1
+ ---
2
+ name: gemini-live-multimodal
3
+ description: Implements real-time bidirectional streaming audio and video pair-programming with the Gemini Live API, WebSocket protocols, ephemeral tokens, and voice activity detection.
4
+ ---
5
+
6
+ # Gemini Live Multimodal (Bidirectional Audio/Visual Streaming)
7
+
8
+ Implements real-time, low-latency, bidirectional audio/video interaction with Google Gemini models (Gemini 2.0 Flash and Gemini Live) using WebSockets, Ephemeral Client Tokens, Voice Activity Detection (VAD), and live screen/DOM streaming.
9
+
10
+ ---
11
+
12
+ ## 1. Architecture: Bidirectional Gemini Live API
13
+
14
+ ```
15
+ ┌─────────────────────────────────────────────────────────────────────────────┐
16
+ │ GEMINI LIVE RUNTIME │
17
+ │ │
18
+ │ ┌─────────────────────┐ Audio (PCM 16kHz) ┌─────────────────────────┐ │
19
+ │ │ Developer Audio │──────────────────────►│ Google GenAI Live API │ │
20
+ │ │ (Microphone Input) │ │ (wss://generativelanguage│ │
21
+ │ └─────────────────────┘ │ .googleapis.com/ws) │ │
22
+ │ │ │ │
23
+ │ ┌─────────────────────┐ Frames (JPEG/WebP) │ • Voice Activity Detect │ │
24
+ │ │ Active Screen / IDE │──────────────────────►│ • Extended Reasoning │ │
25
+ │ │ (Visual Context) │ │ • Sub-second TTFT Voice │ │
26
+ │ └─────────────────────┘ └───────────┬─────────────┘ │
27
+ │ │ │
28
+ │ ┌─────────────────────┐ Synthesized Audio (24kHz) │ │
29
+ │ │ Agent Voice Stream │◄──────────────────────────────────┘ │
30
+ │ │ (Speaker Output) │ │
31
+ │ └─────────────────────┘ │
32
+ └─────────────────────────────────────────────────────────────────────────────┘
33
+ ```
34
+
35
+ ---
36
+
37
+ ## 2. Ephemeral Session Token Minting
38
+
39
+ For enterprise browser/IDE security, never expose primary API keys on clients. Mint short-lived ephemeral tokens from your secure backend:
40
+
41
+ ```python
42
+ import os
43
+ from google import genai
44
+ from google.genai import types
45
+
46
+ client = genai.Client()
47
+
48
+ def create_ephemeral_live_session(user_id: str, ttl_seconds: int = 1800) -> str:
49
+ """Mints an ephemeral session token with pre-configured system instructions."""
50
+ session = client.aio.live.create_session_token(
51
+ model="gemini-2.0-flash-exp",
52
+ config=types.LiveConnectConfig(
53
+ response_modalities=[types.LiveModality.AUDIO],
54
+ system_instruction=types.Content(
55
+ parts=[types.Part.from_text(
56
+ "You are an expert pair-programming agent inside Antigravity. "
57
+ "Analyze screen frames and audio simultaneously. Be concise and actionable."
58
+ )]
59
+ ),
60
+ speech_config=types.SpeechConfig(
61
+ voice_config=types.VoiceConfig(
62
+ prebuilt_voice_config=types.PrebuiltVoiceConfig(voice_name="Aoede")
63
+ )
64
+ )
65
+ )
66
+ )
67
+ return session.token
68
+ ```
69
+
70
+ ---
71
+
72
+ ## 3. Bidirectional WebSocket Audio Streaming (Python Client)
73
+
74
+ Connects to the Gemini Live WebSocket endpoint and streams audio bi-directionally:
75
+
76
+ ```python
77
+ import asyncio
78
+ import websockets
79
+ import json
80
+
81
+ LIVE_WS_URL = "wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1alpha.GenerativeService.BidiGenerateContent"
82
+
83
+ async def stream_live_pair_programming(api_key: str):
84
+ url = f"{LIVE_WS_URL}?key={api_key}"
85
+
86
+ async with websockets.connect(url) as ws:
87
+ # 1. Send initial session handshake
88
+ setup_message = {
89
+ "setup": {
90
+ "model": "models/gemini-2.0-flash-exp",
91
+ "generation_config": {
92
+ "response_modalities": ["AUDIO"],
93
+ "speech_config": {
94
+ "voice_config": {
95
+ "prebuilt_voice_config": {"voice_name": "Puck"}
96
+ }
97
+ }
98
+ }
99
+ }
100
+ }
101
+ await ws.send(json.dumps(setup_message))
102
+
103
+ # 2. Receive live audio chunks & emit to speaker
104
+ async for raw_msg in ws:
105
+ response = json.loads(raw_msg)
106
+ server_content = response.get("serverContent", {})
107
+ model_turn = server_content.get("modelTurn", {})
108
+
109
+ for part in model_turn.get("parts", []):
110
+ inline_data = part.get("inlineData", {})
111
+ if inline_data.get("mimeType", "").startswith("audio/"):
112
+ pcm_chunk = inline_data.get("data")
113
+ # Send to audio output buffer
114
+ ```
115
+
116
+ ---
117
+
118
+ ## 4. Visual Screen Frame Injection
119
+
120
+ Inject captured screen frames at 1 frame per second to ground audio conversation in visual code context:
121
+
122
+ ```python
123
+ import base64
124
+
125
+ async def send_screen_frame(ws, jpeg_bytes: bytes):
126
+ """Encodes a screen capture and sends it as a visual context turn."""
127
+ frame_payload = {
128
+ "realtime_input": {
129
+ "media_chunks": [
130
+ {
131
+ "mime_type": "image/jpeg",
132
+ "data": base64.b64encode(jpeg_bytes).decode("utf-8")
133
+ }
134
+ ]
135
+ }
136
+ }
137
+ await ws.send(json.dumps(frame_payload))
138
+ ```
139
+
140
+ ---
141
+
142
+ ## 5. Enterprise Best Practices & Anti-Patterns
143
+
144
+ | Category | Recommended Pattern | Anti-Pattern |
145
+ | :--- | :--- | :--- |
146
+ | **Authentication** | Ephemeral tokens with 30m TTL minted by backend | Hardcoding permanent API keys in frontend bundles |
147
+ | **Audio Format** | PCM 16-bit little-endian at 16,000Hz mono | Compressed MP3 with variable bitrates causing VAD jitter |
148
+ | **Video Cadence** | Throttled keyframes at 1 FPS or on viewport change | 60 FPS full video ingestion saturating network bandwidth |
149
+ | **Voice Activity** | Enable built-in server-side Voice Activity Detection | Client-side manual push-to-talk creating unnatural latency |