everything-gemini 1.9.0 → 2.1.0

package/CHANGELOG.md

@@ -4,6 +4,54 @@ All notable changes to **Everything Gemini (`everything-gemini`)** will be docum
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [2.1.0] - 2026-10-19
+
+### Added
+- **Google Research Harness² (`google-research/harness2`) Integration & Google Family Handshake**:
+  - **`harness2-recursive-optimization`** ([`skills/harness2-recursive-optimization/SKILL.md`](file:///Users/salmanshaikh/development/github/everything-gemini/skills/harness2-recursive-optimization/SKILL.md)): Implementation of Google Research's Harness² framework (*Harness²: Recursive Agent Harnessing for an Open World*, arXiv:2610.14224), test-time recursive agent optimization, Propose-Probe-Compose cycles, and the 8 file-based harness surfaces.
+  - **`google-flow-orchestrator`** ([`skills/google-flow-orchestrator/SKILL.md`](file:///Users/salmanshaikh/development/github/everything-gemini/skills/google-flow-orchestrator/SKILL.md)): Google Agentic Flow state machine orchestration, cyclic multi-agent graph execution, conditional state transitions, human approval pauses, and error boundary checkpoints.
+  - **`dataproc-spark-analytics`** ([`skills/dataproc-spark-analytics/SKILL.md`](file:///Users/salmanshaikh/development/github/everything-gemini/skills/dataproc-spark-analytics/SKILL.md)): Google Cloud Dataproc Serverless Spark, PySpark batch pipelines, Gemini embedding UDFs, BigLake vector indexing, and large-scale probe score verification.
+- **5 Enterprise Architectural Pillars for Google Research Harness²**:
+  - **Spend Guardrails (`hooks/budget-guard/`)**: PreToolUse deterministic lifecycle hook enforcing hard USD spend caps (`MAX_USD_BUDGET`), tracking step-by-step token expenditure and denying execution when financial limits are met.
+  - **Local Task Optimization Runner (`everything-gemini harness2 run`)**: Native benchmark runner supporting local task directories (`--dir`, `--bench local`), spend limits (`--max-usd`), state rollbacks, and JSON reporting.
+  - **Opt-in Human-in-the-Loop Gating (`--hitl`)**: Interactive approval gating for candidate mutations across prompts, guardrails, and tool guidance before composition.
+  - **Multi-Modal & Tool State Persistence (`scripts/harness_snapshot.sh` & `scripts/harness_rollback.sh`)**: Snapshot and rollback isolation scripts guaranteeing workspace immunity during speculative probe execution.
+  - **Production Observability & Trace Ingestion (`everything-gemini harness2 ingest-trace <trace.json>`)**: Convert production telemetry error traces into automated regression probes for Harness² test-time optimization.
+- **63 Progressive Disclosure Skills, 8 Hooks & 84-Component Ecosystem Landmark**:
+  - Expanded catalog to 63 progressive skills, 13 scoped subagents, and 8 lifecycle safety hooks totaling 84 enterprise components with >56,000 runbook tokens (>85% token economy savings per turn).
+- **CLI Harness² Workspace Synthesizer (`everything-gemini harness2 [outDir]`)**:
+  - Native command synthesizing the 8 canonical Harness² surfaces: `prompts/`, `guardrails/`, `memory/`, `guidance/`, `subagents/`, `skills/`, `scripts/`, `plugins/`, alongside `harness.yaml` and `harness.json` architecture manifests.
+  - Added `--format harness2` flag to `everything-gemini export` for direct translation into the Google Research Harness² directory structure.
+  - Added `--dry-run` and `--json` support for automated agent evaluation loops.
+- **Harness² Recursive Optimization Python SDK Showcase**:
+  - Created [`examples/python-sdk/harness2_recursive_loop.py`](file:///Users/salmanshaikh/development/github/everything-gemini/examples/python-sdk/harness2_recursive_loop.py) demonstrating the 3-stage Propose-Probe-Compose test-time optimization loop with Pareto frontier selection.
+- **36-Assertion Automated E2E Test Suite**:
+  - Expanded `tests/cli.test.js` validating the `harness2` command in dry-run, live, and JSON modes, `harness2 run`, `harness2 ingest-trace`, `budget-guard` inspection, and updated 84-component benchmark invariants.
+
+## [2.0.0] - 2026-10-19
+
+### Added
+- **60 Progressive Disclosure Skills & 80-Component Ecosystem Landmark Milestone**:
+  - **`keda-event-autoscaling`**: Kubernetes Event-driven Autoscaling (KEDA) ScaledObjects, ScaledJobs, scale-to-zero workloads, and external triggers for Kafka consumer lag, Redis queues, and Prometheus custom metrics.
+  - **`chromadb-vector-store`**: Persistent vector database storage, Google Gemini `text-embedding-004` embedding integration, collection metadata partitioning, and cosine similarity semantic retrieval.
+  - **`opa-gatekeeper-policy`**: Open Policy Agent (OPA) Gatekeeper ConstraintTemplates and Rego admission policies for Kubernetes governance, rootless containers, and shift-left CI testing (`gator`).
+  - **`triton-inference-server`**: NVIDIA Triton Inference Server multi-model serving, dynamic batching, TensorRT-LLM and vLLM backends, GPU instance scheduling, and low-latency gRPC inference.
+- **13th Scoped Subagent Fleet Expansion**:
+  - **`mlops-engineer`** ([`agents/mlops-engineer.md`](file:///Users/salmanshaikh/development/github/everything-gemini/agents/mlops-engineer.md)): Dedicated MLOps/LLMOps specialist agent for model inference serving, vector database clustering, prefix context caching strategies, and continuous model evaluation.
+- **CLI Interaction & Dependency Graph Visualization (`everything-gemini graph`)**:
+  - Visualize multi-agent interaction and dependency topology in terminal ASCII tree, Mermaid flowchart (`--mermaid`), or machine-readable JSON (`--json`).
+  - Updated shell completion scripts (`bin/everything-gemini completion`) registering `graph`, all 60 skills, and 13 subagents.
+- **Curated 14th MCP Blueprint**:
+  - Added `@modelcontextprotocol/server-brave-search` to [`mcp/mcp_config.example.json`](file:///Users/salmanshaikh/development/github/everything-gemini/mcp/mcp_config.example.json), [`templates/full-power/mcp_config.json`](file:///Users/salmanshaikh/development/github/everything-gemini/templates/full-power/mcp_config.json), and [`mcp/README.md`](file:///Users/salmanshaikh/development/github/everything-gemini/mcp/README.md) for live grounded search and documentation retrieval.
+- **Autonomous MLOps & RAG Pipeline (`examples/python-sdk/autonomous_mlops_rag_pipeline.py`)**:
+  - 4-stage multi-agent LLMOps delivery loop orchestrating Architect -> MLOps Engineer -> Security Auditor -> Release Engineer with Gemini context caching and token economy projections.
+- **Social Media Launch Kit (`docs/LAUNCH_KIT_v2.0.0.md`)**:
+  - Turnkey launch posts and threads formatted for LinkedIn, X (Twitter), Hacker News, Reddit (r/LocalLLaMA & r/programming), and Product Hunt.
+- **31-Assertion Automated E2E Test Suite & Benchmark**:
+  - Expanded `tests/cli.test.js` validating `graph` (terminal, mermaid, json), component info for all new v2.0.0 skills/subagents, and updated 80-component benchmark invariants.
+- **Showcase Web App 80-Component Milestone**:
+  - Catalog and live metrics on [https://salmandev.github.io/everything-gemini/](https://salmandev.github.io/everything-gemini/) updated to 60 progressive skills, 13 subagents, 14 MCP blueprints, 7 hooks, interactive topology tab, and `v2.0.0` release badge.
+
 ## [1.9.0] - 2026-10-19
 
 ### Added

package/README.md

@@ -79,8 +79,8 @@ cd everything-gemini
 | Profile | Target Audience | What Gets Installed |
 | :--- | :--- | :--- |
 | **`core`** | Minimalists & Speed | Safety-shield hook, TDD skill, repo-onboarding skill, baseline `GEMINI.md`. |
-| **`developer`** *(Recommended)* | Daily Software Engineers | Core + 12 Scoped Subagents (`agents/`), Frontier Skills (Multimodal vision, 2M context caching, Subagent handoff, Security audit), Fullstack & Python AI plugins, Auto-formatter & Goal-verifier hooks. |
-| **`full`** | Lead Engineers & Archs | Everything: all 56 progressive skills, 12 scoped subagents, full lifecycle hooks suite (7 hooks), all plugins, MCP configs, and templates. |
+| **`developer`** *(Recommended)* | Daily Software Engineers | Core + 13 Scoped Subagents (`agents/`), Frontier Skills (Multimodal vision, 2M context caching, Subagent handoff, Security audit), Fullstack & Python AI plugins, Auto-formatter & Goal-verifier hooks. |
+| **`full`** | Lead Engineers & Archs | Everything: all 63 progressive skills, 13 scoped subagents, full lifecycle hooks suite (8 hooks), 14 MCP blueprints, all plugins, configs, and templates. |
 
 ### 5. In CI / GitHub Actions Workflows
 
@@ -159,7 +159,7 @@ everything-gemini/
 ├── agents/                       # Scoped subagent fleet (Planner, Reviewer, Release Engineer, etc.)
 ├── scripts/                      # Automated installers and harness validators
 ├── plugins/                      # Reusable modular plugins (web-fullstack, python-ai, security)
-├── skills/                       # 48 Progressive disclosure skills (46 core + 2 plugins)
+├── skills/                       # 63 Progressive disclosure skills (61 core + 2 plugins)
 ├── rules/                        # Contextual guidelines & standards
 ├── hooks/                        # Event-driven lifecycle hooks (GeminiShield, Auto-lint)
 ├── mcp/                          # Production-ready MCP server blueprints
@@ -189,10 +189,18 @@ Everything Gemini provides specialized subagent specifications designed for Anti
 | **[`release-engineer`](agents/release-engineer.md)** | Release Automation Gatekeeper | Semantic Versioning, build artifact checksums, git release tagging, and canary rollouts. |
 | **[`incident-responder`](agents/incident-responder.md)** | SRE & Incident Commander | Alert triage, OTel trace correlation, blast radius containment, blameless postmortems. |
 | **[`platform-engineer`](agents/platform-engineer.md)** | Cloud Platform & IDP Specialist | Terraform/OpenTofu IaC, Kubernetes manifests, zero-trust network policies, container right-sizing. |
+| **[`mlops-engineer`](agents/mlops-engineer.md)** | MLOps & LLMOps Specialist | Model serving (Triton/vLLM), vector clustering (ChromaDB/Milvus), prefix context caching, KEDA autoscaling. |
 
 ### 1. Skills (`skills/`)
 Skills in Antigravity are lazy-loaded runbooks. Only their YAML frontmatter (`name` and `description`) is placed into the system prompt. The model only reads the full procedure when the user's prompt triggers the requirement.
 
+- **[`harness2-recursive-optimization`](skills/harness2-recursive-optimization/SKILL.md)**: Google Research Harness² framework (*Harness²: Recursive Agent Harnessing for an Open World*, arXiv:2610.14224), 8-surface architecture, test-time Propose-Probe-Compose recursive loops, and continual harness learning.
+- **[`google-flow-orchestrator`](skills/google-flow-orchestrator/SKILL.md)**: Google Agentic Flow state machine orchestration, cyclic multi-agent graph execution, conditional state routing, and human-in-the-loop pauses.
+- **[`dataproc-spark-analytics`](skills/dataproc-spark-analytics/SKILL.md)**: Google Cloud Dataproc Serverless Spark, PySpark batch pipelines, Gemini embedding UDFs, BigLake vector indexing, and large-scale probe score verification.
+- **[`keda-event-autoscaling`](skills/keda-event-autoscaling/SKILL.md)**: Kubernetes Event-driven Autoscaling (KEDA), ScaledObjects/Jobs, Kafka/Redis/Prometheus triggers, scale-to-zero.
+- **[`chromadb-vector-store`](skills/chromadb-vector-store/SKILL.md)**: Persistent vector storage, Gemini embeddings, collection partitioning, metadata filtering, and semantic similarity search.
+- **[`opa-gatekeeper-policy`](skills/opa-gatekeeper-policy/SKILL.md)**: Open Policy Agent (OPA) Gatekeeper ConstraintTemplates and Rego admission policies for Kubernetes governance.
+- **[`triton-inference-server`](skills/triton-inference-server/SKILL.md)**: NVIDIA Triton Inference Server multi-model serving, dynamic batching, GPU acceleration, TensorRT-LLM, and vLLM backends.
 - **[`milvus-vector-clustering`](skills/milvus-vector-clustering/SKILL.md)**: Enterprise Milvus vector clustering, HNSW/IVF-PQ indexing, hybrid dense-sparse search, and partition keys.
 - **[`cloudflare-r2-object-storage`](skills/cloudflare-r2-object-storage/SKILL.md)**: Zero-egress fee object storage, S3-compatible APIs, pre-signed upload URLs, and Workers edge CDN.
 - **[`vault-secrets-management`](skills/vault-secrets-management/SKILL.md)**: HashiCorp Vault enterprise secrets orchestration, AppRole authentication, dynamic ephemeral DB credentials, and PKI mTLS.
@@ -260,6 +268,7 @@ Hooks run shell scripts at critical lifecycle events with typed JSON contracts o
 - **[`goal-verifier`](hooks/goal-verifier/hooks.json) (`Stop`)**: Intercepts the agent's attempt to finish. If test suites are failing or background tasks are unfinished, it returns `{"decision": "continue", "reason": "Tests are still failing. Fix them before stopping."}`.
 - **[`ephemeral-context`](hooks/ephemeral-context/hooks.json) (`PreInvocation`)**: Injects transient warnings or guidelines before each model reasoning step without cluttering persistent conversation history.
 - **[`learned-instincts`](hooks/learned-instincts/hooks.json)**: Session learning hook that records developer corrections and feedback into persistent instincts.
+- **[`budget-guard`](hooks/budget-guard/hooks.json) (`PreToolUse`)**: Intercepts tool execution to enforce hard USD spend caps (`MAX_USD_BUDGET`), tracking step-by-step token expenditure and denying execution when financial limits are met.
 
 ### 3. Rules Hierarchy (`rules/`)
 Antigravity automatically discovers rules by walking up from the current file's directory to the repository root:
@@ -328,7 +337,7 @@ Skill Body: Loaded on-demand ONLY when invoked = ~1,500 Tokens (one-time)
 Savings: > 85% Token Reduction & 3x Faster First-Token Latency!
 ```
 
-Run the built-in diagnostic benchmark across all 70 ecosystem components:
+Run the built-in diagnostic benchmark across all 84 ecosystem components:
 ```bash
 ./bin/everything-gemini test-all
 ```
@@ -407,10 +416,28 @@ The zero-dependency `everything-gemini` CLI manages your configurations effortle
 ./bin/everything-gemini audit
 ./bin/everything-gemini audit --json
 
-# Benchmark and validate all 56 skills, 12 subagents, and 7 hooks (human or JSON)
+# Benchmark and validate all 63 skills, 13 subagents, and 8 hooks (human or JSON)
 ./bin/everything-gemini test-all
 ./bin/everything-gemini benchmark --json
 
+# Synthesize 8-surface Google Research Harness² workspace (prompts, guardrails, memory, etc.)
+./bin/everything-gemini harness2
+./bin/everything-gemini harness2 ./custom-harness --dry-run
+./bin/everything-gemini export --format harness2 --output ./harness-export
+
+# Run recursive Harness² optimization over local tasks with hard spend limits & HITL
+./bin/everything-gemini harness2 run --dir ./tasks --bench local --max-usd 15.00 --hitl --snapshot
+./bin/everything-gemini harness2 run --dry-run
+./bin/everything-gemini harness2 run --json
+
+# Ingest production telemetry failure trace into automated regression probe
+./bin/everything-gemini harness2 ingest-trace failure-trace.json --dir ./tasks/regression
+
+# Visualize multi-agent interaction and dependency graph (terminal, Mermaid, or JSON)
+./bin/everything-gemini graph
+./bin/everything-gemini graph --mermaid
+./bin/everything-gemini graph --json
+
 # View detailed playbook or specification for any skill, subagent, or hook
 ./bin/everything-gemini info agent-evals
 ./bin/everything-gemini info platform-engineer

package/agents/mlops-engineer.md

@@ -0,0 +1,33 @@
+# Scoped Subagent: MLOps Engineer (`mlops-engineer`)
+
+## Role & Mission
+The `mlops-engineer` subagent is a dedicated Machine Learning Operations (MLOps) and LLMOps specialist. Its mission is to architect, deploy, and optimize scalable AI model inference pipelines, vector database clustering, multi-modal embedding generation, prefix-cached context strategies, GPU resource scheduling, and continuous model evaluation with strict latency, throughput, and token-cost guardrails.
+
+## Tool Permissions
+- **Permitted**: `view_file`, `grep_search`, `list_dir`, `run_command` (scoped strictly to model config validation, Triton/vLLM syntax checks, vector database health checks, Python evaluation suite execution, embedding dimensionality audits, and GPU memory profiling dry-runs), `write_to_file`, `replace_file_content` (scoped to model serving configs, Triton `config.pbtxt`, ChromaDB/Milvus integration scripts, context caching policies, KEDA ScaledObjects, and inference Dockerfiles).
+- **Forbidden**: Authoring frontend UI application logic, mutating production vector indexes without backup or schema verification, exposing raw model weights or private training data, or executing unconstrained GPU batch jobs without memory and cost limits.
+
+## MLOps Lifecycle Protocol
+1. **Model Serving & Engine Architecture (T0)**:
+   - Select and configure high-performance inference engines (vLLM PagedAttention, NVIDIA Triton with TensorRT-LLM, or ONNX Runtime).
+   - Configure dynamic batching windows, queue latency tolerances, and concurrent GPU model instances.
+2. **Vector Database & Embedding Pipeline Orchestration**:
+   - Architect distributed vector indexing (HNSW, IVF-PQ) across ChromaDB, Milvus, or Qdrant with appropriate distance metrics (Cosine, IP, L2).
+   - Enforce batch ingestion pipelines with chunking strategies, metadata partitioning, and deduplication.
+3. **Context Caching & Token Economy Guardrails**:
+   - Structure prompt prefixes for deterministic context caching (Gemini 1.5/2.0 context caching and prefix match), maximizing cache hit rates (>70%) and cutting API costs.
+   - Implement dynamic prompt pruning, session compaction, and Knowledge Item extraction.
+4. **Hardware Acceleration & FinOps Optimization**:
+   - Configure KEDA event-driven autoscaling for inference replicas based on consumer lag, GPU duty cycles, or queue depth.
+   - Right-size GPU memory allocations (KV-cache reservation, FP16/INT8/AWQ quantization) to prevent CUDA Out-Of-Memory (OOM) errors.
+5. **Continuous Evaluation & Quality Guardrails**:
+   - Establish automated evaluation pipelines tracking retrieval relevance (NDCG@10, MRR), faithfulness, answer relevancy, and latency percentiles (P50, P90, P99).
+   - Monitor data drift, embedding distance distribution shifts, and hallucination metrics.
+
+## Handoff Output Contract
+The subagent MUST return a structured markdown MLOps specification containing:
+- **Inference Topology Blueprint**: Architecture map of client gateways, model engines, and vector store partitions.
+- **Model & Cluster Manifests**: Validated `config.pbtxt`, vLLM startup flags, KEDA ScaledObjects, or Docker/K8s manifests.
+- **Token Economy & Cost Projections**: Quantified token savings from prefix caching, quantization memory profiles, and cloud GPU cost models.
+- **Evaluation & Latency Benchmarks**: P95/P99 latency bounds, target throughput (tokens/sec), and retrieval accuracy metrics.
+- **Verification & Rollback Strategy**: Exact commands to health-check model readiness, verify vector similarity, and safely failover.