everything-gemini 1.8.0 → 2.0.0

package/CHANGELOG.md

@@ -4,6 +4,53 @@ All notable changes to **Everything Gemini (`everything-gemini`)** will be docum
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [2.0.0] - 2026-10-19
+
+### Added
+- **60 Progressive Disclosure Skills & 80-Component Ecosystem Landmark Milestone**:
+  - **`keda-event-autoscaling`**: Kubernetes Event-driven Autoscaling (KEDA) ScaledObjects, ScaledJobs, scale-to-zero workloads, and external triggers for Kafka consumer lag, Redis queues, and Prometheus custom metrics.
+  - **`chromadb-vector-store`**: Persistent vector database storage, Google Gemini `text-embedding-004` embedding integration, collection metadata partitioning, and cosine similarity semantic retrieval.
+  - **`opa-gatekeeper-policy`**: Open Policy Agent (OPA) Gatekeeper ConstraintTemplates and Rego admission policies for Kubernetes governance, rootless containers, and shift-left CI testing (`gator`).
+  - **`triton-inference-server`**: NVIDIA Triton Inference Server multi-model serving, dynamic batching, TensorRT-LLM and vLLM backends, GPU instance scheduling, and low-latency gRPC inference.
+- **13th Scoped Subagent Fleet Expansion**:
+  - **`mlops-engineer`** ([`agents/mlops-engineer.md`](file:///Users/salmanshaikh/development/github/everything-gemini/agents/mlops-engineer.md)): Dedicated MLOps/LLMOps specialist agent for model inference serving, vector database clustering, prefix context caching strategies, and continuous model evaluation.
+- **CLI Interaction & Dependency Graph Visualization (`everything-gemini graph`)**:
+  - Visualize multi-agent interaction and dependency topology in terminal ASCII tree, Mermaid flowchart (`--mermaid`), or machine-readable JSON (`--json`).
+  - Updated shell completion scripts (`bin/everything-gemini completion`) registering `graph`, all 60 skills, and 13 subagents.
+- **Curated 14th MCP Blueprint**:
+  - Added `@modelcontextprotocol/server-brave-search` to [`mcp/mcp_config.example.json`](file:///Users/salmanshaikh/development/github/everything-gemini/mcp/mcp_config.example.json), [`templates/full-power/mcp_config.json`](file:///Users/salmanshaikh/development/github/everything-gemini/templates/full-power/mcp_config.json), and [`mcp/README.md`](file:///Users/salmanshaikh/development/github/everything-gemini/mcp/README.md) for live grounded search and documentation retrieval.
+- **Autonomous MLOps & RAG Pipeline (`examples/python-sdk/autonomous_mlops_rag_pipeline.py`)**:
+  - 4-stage multi-agent LLMOps delivery loop orchestrating Architect -> MLOps Engineer -> Security Auditor -> Release Engineer with Gemini context caching and token economy projections.
+- **Social Media Launch Kit (`docs/LAUNCH_KIT_v2.0.0.md`)**:
+  - Turnkey launch posts and threads formatted for LinkedIn, X (Twitter), Hacker News, Reddit (r/LocalLLaMA & r/programming), and Product Hunt.
+- **31-Assertion Automated E2E Test Suite & Benchmark**:
+  - Expanded `tests/cli.test.js` validating `graph` (terminal, mermaid, json), component info for all new v2.0.0 skills/subagents, and updated 80-component benchmark invariants.
+- **Showcase Web App 80-Component Milestone**:
+  - Catalog and live metrics on [https://salmandev.github.io/everything-gemini/](https://salmandev.github.io/everything-gemini/) updated to 60 progressive skills, 13 subagents, 14 MCP blueprints, 7 hooks, interactive topology tab, and `v2.0.0` release badge.
+
+## [1.9.0] - 2026-10-19
+
+### Added
+- **56 Progressive Disclosure Skills & 75-Component Ecosystem Milestone**:
+  - **`milvus-vector-clustering`**: Enterprise Milvus vector clustering, HNSW vs. IVF-PQ indexing tradeoffs, hybrid dense-sparse search (BM25 + cosine), dynamic schema partition keys, and scalar metadata filtering.
+  - **`cloudflare-r2-object-storage`**: Zero-egress fee object storage, S3-compatible APIs, pre-signed upload URL delegation, multipart parallel streaming uploads, and Workers edge CDN caching.
+  - **`vault-secrets-management`**: HashiCorp Vault enterprise secrets orchestration, AppRole machine-to-machine authentication, dynamic ephemeral database credentials with auto-revocation, and PKI mTLS certificate rotation.
+  - **`distributed-rate-limiting`**: High-throughput distributed rate limiting architectures, Redis Lua script atomic sliding window logs, token bucket replenishment, tiered tenant quotas, and RFC 6585 compliance.
+- **12th Scoped Subagent Fleet Expansion**:
+  - **`platform-engineer`** ([`agents/platform-engineer.md`](file:///Users/salmanshaikh/development/github/everything-gemini/agents/platform-engineer.md)): Dedicated cloud platform, Infrastructure as Code (IaC), and Internal Developer Platform (IDP) specialist enforcing multi-zone Kubernetes topology, container resource right-sizing, zero-trust network boundaries, and developer self-service golden paths.
+- **CLI Workspace Synchronization & Unified Component Scaffolding**:
+  - **`everything-gemini sync`**: Synchronize installed workspace skills, subagents, and hooks against the latest catalog, verifying checksum integrity and auto-updating stale components with `--dry-run` and `--json` support.
+  - **`everything-gemini scaffold <skill|agent|hook> <name>`** (aliases `new`, `create`): Scaffold 100% compliant progressive disclosure skills, scoped subagent specifications, and lifecycle hooks with YAML frontmatter, reference schemas, and executable helper scripts.
+  - Expanded shell completion scripts (`bin/everything-gemini completion`) for Bash and Zsh to autocomplete `sync`, `scaffold`, all 56 skills, and 12 subagents.
+- **Curated MCP Puppeteer Blueprint**:
+  - Added `@modelcontextprotocol/server-puppeteer` to [`mcp/mcp_config.example.json`](file:///Users/salmanshaikh/development/github/everything-gemini/mcp/mcp_config.example.json), [`templates/full-power/mcp_config.json`](file:///Users/salmanshaikh/development/github/everything-gemini/templates/full-power/mcp_config.json), and [`mcp/README.md`](file:///Users/salmanshaikh/development/github/everything-gemini/mcp/README.md) for headless browser automation and visual testing.
+- **Cloud Platform Provisioning Pipeline (`examples/python-sdk/platform_provisioning_pipeline.py`)**:
+  - 4-stage multi-agent infrastructure delivery loop orchestrating Architect -> Platform Engineer -> Security Auditor -> Release Engineer using the Python SDK.
+- **29-Assertion Automated E2E Test Suite & Benchmark**:
+  - Expanded `tests/cli.test.js` validating the new `sync` and `scaffold` commands (dry-run and live modes), component inspections for new skills/subagents, and updated 75-component benchmark invariants.
+- **Showcase Web App 75-Component Expansion**:
+  - Catalog and live metrics on [https://salmandev.github.io/everything-gemini/](https://salmandev.github.io/everything-gemini/) updated to 56 progressive skills, 12 subagents, 7 hooks, and `v1.9.0` release badge.
+
 ## [1.8.0] - 2026-10-19
 
 ### Added

package/CONTRIBUTING.md

@@ -163,10 +163,15 @@ Looking for inspiration on what to contribute? The community has requested the f
 | **`debezium-cdc`** | Data Eng | Change Data Capture (CDC) streaming from PostgreSQL / MySQL to Kafka / PubSub | ✅ *Shipped in v1.8.0* |
 | **`cilium-ebpf-networking`** | Cloud Native | eBPF-based service mesh, L7 security policies, and transparent encryption | ✅ *Shipped in v1.8.0* |
 | **`litellm-proxy-gateway`** | AI Gateway | Unified OpenAI-compatible multi-provider routing, fallback budgets, and load balancing | ✅ *Shipped in v1.8.0* |
+| **`milvus-vector-clustering`** | Vector DB | High-scale vector clustering, HNSW/IVF-PQ indexing, and hybrid dense-sparse search | ✅ *Shipped in v1.9.0* |
+| **`cloudflare-r2-object-storage`** | Storage | S3-compatible zero-egress object storage, pre-signed upload URLs, and Workers CDN | ✅ *Shipped in v1.9.0* |
+| **`vault-secrets-management`** | Security | HashiCorp Vault enterprise secrets orchestration, dynamic DB roles, and PKI mTLS | ✅ *Shipped in v1.9.0* |
+| **`distributed-rate-limiting`** | Traffic | Redis sliding-window logs, token buckets, tiered quotas, and RFC 6585 compliance | ✅ *Shipped in v1.9.0* |
 | **`keda-autoscaling`** | Cloud Native | Event-driven Kubernetes workload autoscaling (Kafka, SQS, RabbitMQ, Redis) | 🚀 *Open for PR* |
 | **`chromadb-vector-store`** | AI Eng | Embedded and client/server vector embeddings, HNSW indexing, and collection filters | 🚀 *Open for PR* |
 | **`opa-gatekeeper`** | Policy | Open Policy Agent (OPA) admission control, Rego constraints, and pod security standards | 🚀 *Open for PR* |
 | **`fal-ai-media`** | Generative AI | Ultra-fast multimodal diffusion, video generation, and real-time inference endpoints | 🚀 *Open for PR* |
+| **`iceberg-lakehouse`** | Data Lake | Apache Iceberg table format, ACID metadata, hidden partitioning, and time-travel | 🚀 *Open for PR* |
 
 ### How to Build a Wishlist Item:
 1. Open an issue using the [Skill Proposal template](.github/ISSUE_TEMPLATE/skill_proposal.md).

package/README.md

@@ -79,8 +79,8 @@ cd everything-gemini
 | Profile | Target Audience | What Gets Installed |
 | :--- | :--- | :--- |
 | **`core`** | Minimalists & Speed | Safety-shield hook, TDD skill, repo-onboarding skill, baseline `GEMINI.md`. |
-| **`developer`** *(Recommended)* | Daily Software Engineers | Core + 11 Scoped Subagents (`agents/`), Frontier Skills (Multimodal vision, 2M context caching, Subagent handoff, Security audit), Fullstack & Python AI plugins, Auto-formatter & Goal-verifier hooks. |
-| **`full`** | Lead Engineers & Archs | Everything: all 52 progressive skills, 11 scoped subagents, full lifecycle hooks suite (7 hooks), all plugins, MCP configs, and templates. |
+| **`developer`** *(Recommended)* | Daily Software Engineers | Core + 13 Scoped Subagents (`agents/`), Frontier Skills (Multimodal vision, 2M context caching, Subagent handoff, Security audit), Fullstack & Python AI plugins, Auto-formatter & Goal-verifier hooks. |
+| **`full`** | Lead Engineers & Archs | Everything: all 60 progressive skills, 13 scoped subagents, full lifecycle hooks suite (7 hooks), 14 MCP blueprints, all plugins, configs, and templates. |
 
 ### 5. In CI / GitHub Actions Workflows
 
@@ -159,7 +159,7 @@ everything-gemini/
 ├── agents/                       # Scoped subagent fleet (Planner, Reviewer, Release Engineer, etc.)
 ├── scripts/                      # Automated installers and harness validators
 ├── plugins/                      # Reusable modular plugins (web-fullstack, python-ai, security)
-├── skills/                       # 48 Progressive disclosure skills (46 core + 2 plugins)
+├── skills/                       # 60 Progressive disclosure skills (58 core + 2 plugins)
 ├── rules/                        # Contextual guidelines & standards
 ├── hooks/                        # Event-driven lifecycle hooks (GeminiShield, Auto-lint)
 ├── mcp/                          # Production-ready MCP server blueprints
@@ -188,10 +188,20 @@ Everything Gemini provides specialized subagent specifications designed for Anti
 | **[`documentation-writer`](agents/documentation-writer.md)** | Technical Documentation Specialist | API specifications, ADR governance, changelogs, and clickable file:// link fidelity. |
 | **[`release-engineer`](agents/release-engineer.md)** | Release Automation Gatekeeper | Semantic Versioning, build artifact checksums, git release tagging, and canary rollouts. |
 | **[`incident-responder`](agents/incident-responder.md)** | SRE & Incident Commander | Alert triage, OTel trace correlation, blast radius containment, blameless postmortems. |
+| **[`platform-engineer`](agents/platform-engineer.md)** | Cloud Platform & IDP Specialist | Terraform/OpenTofu IaC, Kubernetes manifests, zero-trust network policies, container right-sizing. |
+| **[`mlops-engineer`](agents/mlops-engineer.md)** | MLOps & LLMOps Specialist | Model serving (Triton/vLLM), vector clustering (ChromaDB/Milvus), prefix context caching, KEDA autoscaling. |
 
 ### 1. Skills (`skills/`)
 Skills in Antigravity are lazy-loaded runbooks. Only their YAML frontmatter (`name` and `description`) is placed into the system prompt. The model only reads the full procedure when the user's prompt triggers the requirement.
 
+- **[`keda-event-autoscaling`](skills/keda-event-autoscaling/SKILL.md)**: Kubernetes Event-driven Autoscaling (KEDA), ScaledObjects/Jobs, Kafka/Redis/Prometheus triggers, scale-to-zero.
+- **[`chromadb-vector-store`](skills/chromadb-vector-store/SKILL.md)**: Persistent vector storage, Gemini embeddings, collection partitioning, metadata filtering, and semantic similarity search.
+- **[`opa-gatekeeper-policy`](skills/opa-gatekeeper-policy/SKILL.md)**: Open Policy Agent (OPA) Gatekeeper ConstraintTemplates and Rego admission policies for Kubernetes governance.
+- **[`triton-inference-server`](skills/triton-inference-server/SKILL.md)**: NVIDIA Triton Inference Server multi-model serving, dynamic batching, GPU acceleration, TensorRT-LLM, and vLLM backends.
+- **[`milvus-vector-clustering`](skills/milvus-vector-clustering/SKILL.md)**: Enterprise Milvus vector clustering, HNSW/IVF-PQ indexing, hybrid dense-sparse search, and partition keys.
+- **[`cloudflare-r2-object-storage`](skills/cloudflare-r2-object-storage/SKILL.md)**: Zero-egress fee object storage, S3-compatible APIs, pre-signed upload URLs, and Workers edge CDN.
+- **[`vault-secrets-management`](skills/vault-secrets-management/SKILL.md)**: HashiCorp Vault enterprise secrets orchestration, AppRole authentication, dynamic ephemeral DB credentials, and PKI mTLS.
+- **[`distributed-rate-limiting`](skills/distributed-rate-limiting/SKILL.md)**: High-throughput distributed rate limiting using Redis sliding window logs, token buckets, and RFC 6585 compliance.
 - **[`vllm-inference`](skills/vllm-inference/SKILL.md)**: High-throughput LLM serving, PagedAttention KV cache, continuous batching, and tensor parallelism.
 - **[`debezium-cdc`](skills/debezium-cdc/SKILL.md)**: Change Data Capture (CDC) streaming, PostgreSQL logical decoding, Kafka Connect, and outbox patterns.
 - **[`cilium-ebpf-networking`](skills/cilium-ebpf-networking/SKILL.md)**: Cloud-native eBPF networking, transparent mTLS, L7 CiliumNetworkPolicies, and Hubble observability.
@@ -402,20 +412,29 @@ The zero-dependency `everything-gemini` CLI manages your configurations effortle
 ./bin/everything-gemini audit
 ./bin/everything-gemini audit --json
 
-# Benchmark and validate all 52 skills, 11 subagents, and 7 hooks (human or JSON)
+# Benchmark and validate all 60 skills, 13 subagents, and 7 hooks (human or JSON)
 ./bin/everything-gemini test-all
 ./bin/everything-gemini benchmark --json
 
+# Visualize multi-agent interaction and dependency graph (terminal, Mermaid, or JSON)
+./bin/everything-gemini graph
+./bin/everything-gemini graph --mermaid
+./bin/everything-gemini graph --json
+
 # View detailed playbook or specification for any skill, subagent, or hook
 ./bin/everything-gemini info agent-evals
-./bin/everything-gemini info planner
+./bin/everything-gemini info platform-engineer
 ./bin/everything-gemini info gemini-shield
 
-# Scaffold a new progressive-disclosure skill
-./bin/everything-gemini new-skill my-custom-skill
+# Synchronize workspace with latest catalog & verify integrity
+./bin/everything-gemini sync
+./bin/everything-gemini sync --dry-run
+./bin/everything-gemini sync --json
 
-# Scaffold a new scoped subagent
-./bin/everything-gemini new-agent my-worker-agent
+# Scaffold a new progressive-disclosure skill, scoped subagent, or lifecycle hook
+./bin/everything-gemini scaffold skill my-custom-skill
+./bin/everything-gemini scaffold agent my-worker-agent
+./bin/everything-gemini scaffold hook my-safety-hook
 
 # Generate shell autocompletion script (bash/zsh)
 ./bin/everything-gemini completion >> ~/.zshrc

package/agents/mlops-engineer.md

@@ -0,0 +1,33 @@
+# Scoped Subagent: MLOps Engineer (`mlops-engineer`)
+
+## Role & Mission
+The `mlops-engineer` subagent is a dedicated Machine Learning Operations (MLOps) and LLMOps specialist. Its mission is to architect, deploy, and optimize scalable AI model inference pipelines, vector database clustering, multi-modal embedding generation, prefix-cached context strategies, GPU resource scheduling, and continuous model evaluation with strict latency, throughput, and token-cost guardrails.
+
+## Tool Permissions
+- **Permitted**: `view_file`, `grep_search`, `list_dir`, `run_command` (scoped strictly to model config validation, Triton/vLLM syntax checks, vector database health checks, Python evaluation suite execution, embedding dimensionality audits, and GPU memory profiling dry-runs), `write_to_file`, `replace_file_content` (scoped to model serving configs, Triton `config.pbtxt`, ChromaDB/Milvus integration scripts, context caching policies, KEDA ScaledObjects, and inference Dockerfiles).
+- **Forbidden**: Authoring frontend UI application logic, mutating production vector indexes without backup or schema verification, exposing raw model weights or private training data, or executing unconstrained GPU batch jobs without memory and cost limits.
+
+## MLOps Lifecycle Protocol
+1. **Model Serving & Engine Architecture (T0)**:
+   - Select and configure high-performance inference engines (vLLM PagedAttention, NVIDIA Triton with TensorRT-LLM, or ONNX Runtime).
+   - Configure dynamic batching windows, queue latency tolerances, and concurrent GPU model instances.
+2. **Vector Database & Embedding Pipeline Orchestration**:
+   - Architect distributed vector indexing (HNSW, IVF-PQ) across ChromaDB, Milvus, or Qdrant with appropriate distance metrics (Cosine, IP, L2).
+   - Enforce batch ingestion pipelines with chunking strategies, metadata partitioning, and deduplication.
+3. **Context Caching & Token Economy Guardrails**:
+   - Structure prompt prefixes for deterministic context caching (Gemini 1.5/2.0 context caching and prefix match), maximizing cache hit rates (>70%) and cutting API costs.
+   - Implement dynamic prompt pruning, session compaction, and Knowledge Item extraction.
+4. **Hardware Acceleration & FinOps Optimization**:
+   - Configure KEDA event-driven autoscaling for inference replicas based on consumer lag, GPU duty cycles, or queue depth.
+   - Right-size GPU memory allocations (KV-cache reservation, FP16/INT8/AWQ quantization) to prevent CUDA Out-Of-Memory (OOM) errors.
+5. **Continuous Evaluation & Quality Guardrails**:
+   - Establish automated evaluation pipelines tracking retrieval relevance (NDCG@10, MRR), faithfulness, answer relevancy, and latency percentiles (P50, P90, P99).
+   - Monitor data drift, embedding distance distribution shifts, and hallucination metrics.
+
+## Handoff Output Contract
+The subagent MUST return a structured markdown MLOps specification containing:
+- **Inference Topology Blueprint**: Architecture map of client gateways, model engines, and vector store partitions.
+- **Model & Cluster Manifests**: Validated `config.pbtxt`, vLLM startup flags, KEDA ScaledObjects, or Docker/K8s manifests.
+- **Token Economy & Cost Projections**: Quantified token savings from prefix caching, quantization memory profiles, and cloud GPU cost models.
+- **Evaluation & Latency Benchmarks**: P95/P99 latency bounds, target throughput (tokens/sec), and retrieval accuracy metrics.
+- **Verification & Rollback Strategy**: Exact commands to health-check model readiness, verify vector similarity, and safely failover.

package/agents/platform-engineer.md

@@ -0,0 +1,33 @@
+# Scoped Subagent: Platform Engineer (`platform-engineer`)
+
+## Role & Mission
+The `platform-engineer` subagent is a dedicated cloud platform, Infrastructure as Code (IaC), and Internal Developer Platform (IDP) specialist. Its mission is to design, provision, and maintain production-grade cloud topology, Kubernetes manifests, Helm charts, Terraform/OpenTofu modules, zero-trust network boundaries, secret rotation integrations, and developer self-service golden paths with automated cost and reliability guardrails.
+
+## Tool Permissions
+- **Permitted**: `view_file`, `grep_search`, `list_dir`, `run_command` (scoped strictly to IaC syntax validation, `terraform plan`/`tofu plan`, `helm lint`/`helm template`, `docker build` dry-runs, and infrastructure policy scanning via Trivy/Checkov), `write_to_file`, `replace_file_content` (scoped to Terraform files, Kubernetes manifests, Helm values, Dockerfiles, and CI/CD platform templates).
+- **Forbidden**: Authoring core application frontend/UI code, running direct destructive production infrastructure destruction (`terraform destroy` without explicit human signoff), disabling cluster-wide RBAC policies, or checking unencrypted credentials into version control.
+
+## Platform Engineering Lifecycle Protocol
+1. **Infrastructure Architecture & Topology (T0)**:
+   - Define multi-zone/multi-region high-availability topology, VPC subnet boundaries, and Kubernetes namespace isolation.
+   - Establish container resource quotas (`requests`/`limits`), Horizontal Pod Autoscalers (HPA), and PodDisruptionBudgets (PDB).
+2. **Infrastructure as Code (IaC) Provisioning**:
+   - Write modular, reusable Terraform / OpenTofu configurations adhering to clean state management (remote S3/GCS state backends with DynamoDB/Postgres locks).
+   - Package services into parameterized Helm charts or Kustomize overlays for dev, staging, and production environments.
+3. **Zero-Trust Security & Secrets Integration**:
+   - Configure workload identity federation (IAM roles for service accounts / IRSA) to eliminate static cloud credentials.
+   - Integrate Vault Agent sidecars or External Secrets Operator for dynamic secret injection and automatic rotation.
+4. **Reliability & Traffic Guardrails**:
+   - Configure ingress controllers, Cilium eBPF network policies, and distributed rate limiting to guard cluster entry points.
+   - Define health check probes (`livenessProbe`, `readinessProbe`, `startupProbe`) with sensible timeout tolerances.
+5. **FinOps & Resource Right-Sizing**:
+   - Apply CPU/memory right-sizing recommendations based on historical usage metrics to prevent over-provisioning and CPU throttling.
+   - Enforce resource labeling and cost-allocation tagging (`CostCenter`, `Environment`, `Owner`, `ManagedBy`).
+
+## Handoff Output Contract
+The subagent MUST return a structured markdown infrastructure specification containing:
+- **Topology Blueprint**: Visual or structured description of compute, storage, networking, and service mesh boundaries.
+- **IaC & Manifest Deliverables**: Validated Terraform/Helm/Kubernetes manifests ready for automated deployment.
+- **Security & Compliance Controls**: Verification of IAM least-privilege roles, network policies, and secret injection mechanisms.
+- **Resource Sizing & Cost Impact**: Specified CPU/RAM limits, autoscaling parameters, and estimated monthly cloud expenditure impact.
+- **Verification & Rollback Strategy**: Concrete health validation commands and an automated rollback procedure.