duoops 0.1.9 → 0.2.1

package/README.md

@@ -1,35 +1,50 @@
 # DuoOps
 
-> **Toolset for Explainable and Sustainable CI on GitLab.**
+> **Sustainable and Explainable CI/CD for GitLab — powered by AI.**
 
-DuoOps is a developer-focused CLI and Web Portal designed to make GitLab CI/CD pipelines more transparent, debuggable, and environmentally sustainable. It leverages AI agents to explain failures and provides tools to measure and visualize the carbon footprint of your CI jobs.
+DuoOps helps developers understand, debug, and reduce the environmental impact of their GitLab CI/CD pipelines. It combines AI-powered troubleshooting with carbon footprint measurement, giving teams actionable insights to build greener software.
 
 ![License](https://img.shields.io/badge/license-MIT-blue.svg)
-![Version](https://img.shields.io/badge/version-0.0.0-green.svg)
+![GitLab Hackathon 2026](https://img.shields.io/badge/GitLab_Hackathon-2026-orange.svg)
+
+---
+
+<p align="center">
+  <img src="hackathon-files/deliverables.png" alt="DuoOps Deliverables — CI Component, MCP Server, CLI Tool" width="700" />
+</p>
+
+| Deliverable | Description |
+| :--- | :--- |
+| **CI Component** | Drop-in GitLab CI component that measures carbon emissions per job using CPU/RAM timeseries, then persists results to BigQuery. |
+| **MCP Server** | Model Context Protocol server that exposes emissions data to AI agents (GitLab Duo Chat, Claude, etc.) via standardized tools. |
+| **CLI Tool** | Local command-line interface for pipeline inspection, AI-powered troubleshooting, carbon measurement, and launching the web portal. |
+
+---
 
 ## 🌟 Key Features
 
-*   **🤖 AI-Powered Troubleshooting**: Ask questions about your pipelines in plain English. The AI agent (powered by GitLab Duo or Google Vertex AI) analyzes logs, identifies root causes, and suggests fixes.
-*   **🔍 Deep Pipeline Inspection**: View pipeline status, job details, and logs directly from your terminal.
-*   **🌱 Carbon Measurement**: Calculate the carbon emissions of your CI jobs based on CPU/RAM usage timeseries. Track your environmental impact over time.
-*   **📊 Web Portal**: A local React-based dashboard to visualize pipeline metrics and chat with the AI assistant in a rich interface.
-*   **🧠 Intelligent Summarization**: Automatically summarizes long job logs using sub-agents to provide concise failure analysis without token limit issues.
+- **🤖 AI-Powered Troubleshooting** — Ask questions about your pipelines in plain English. The AI agent (GitLab Duo or Vertex AI) analyzes logs, identifies root causes, and suggests fixes.
+- **🌱 Carbon Measurement** — Calculate CO₂ emissions of CI jobs based on CPU/RAM usage timeseries. Track your environmental impact over time.
+- **🔌 MCP Integration** — Connect DuoOps to GitLab Duo Chat so your AI assistant can query emissions data directly.
+- **📊 Web Portal** — A local React dashboard (built with [Palantir Blueprint.js](https://blueprintjs.com/)) to visualize metrics, manage budgets, and chat with the AI assistant.
+- **🔍 Deep Pipeline Inspection** — View pipeline status, job details, and logs from your terminal.
+- **🧠 Intelligent Summarization** — Automatically summarizes long job logs using sub-agents for concise failure analysis.
+
+---
 
 ## 🚀 Quick Start
 
 ### Prerequisites
 
-*   **Node.js** >= 18.0.0
-*   **GitLab Account**: A Personal Access Token with `api` scope.
-*   **(Optional) Google Cloud SDK**: Required only for BigQuery metrics.
-    *   Install the `gcloud` CLI.
-    *   Authenticate: `gcloud auth application-default login`
+- **Node.js** >= 18.0.0
+- **GitLab Account** with a Personal Access Token (`api` scope)
+- *(Optional)* **Google Cloud SDK** — for BigQuery metrics and MCP server deployment
 
 ### Installation
 
 ```bash
 # Clone the repository
-git clone https://github.com/youneslaaroussi/duoops.git
+git clone https://gitlab.com/youneslaaroussi/duoops.git
 cd duoops
 
 # Install dependencies
@@ -44,59 +59,48 @@ npm link
 
 ### Configuration
 
-Run the interactive initialization command to set up your credentials:
+Run the interactive setup:
 
 ```bash
 duoops init
 ```
 
-This will prompt you for:
-*   GitLab URL (default: `https://gitlab.com`)
-*   GitLab Personal Access Token
-*   (Optional) BigQuery settings for sustainability tracking
+This prompts you for your GitLab URL, Personal Access Token, and optional BigQuery settings.
 
-Alternatively, you can configure DuoOps using environment variables:
+Or use environment variables:
 
 | Variable | Description |
 | :--- | :--- |
-| `GITLAB_TOKEN` | Your GitLab Personal Access Token |
+| `GITLAB_TOKEN` | GitLab Personal Access Token |
 | `GITLAB_URL` | GitLab instance URL (default: `https://gitlab.com`) |
-| `DUOOPS_AI_PROVIDER` | AI Provider: `gitlab` (default) or `vertex` |
-| `GCP_PROJECT_ID` | Google Cloud Project ID (required for Vertex AI) |
-| `GCP_LOCATION` | Google Cloud Location (default: `us-central1`) |
+| `DUOOPS_AI_PROVIDER` | AI provider: `gitlab` (default) or `vertex` |
+| `GCP_PROJECT_ID` | Google Cloud Project ID (for Vertex AI / BigQuery) |
+| `GCP_LOCATION` | Google Cloud location (default: `us-central1`) |
 
-## 📖 Usage
+---
 
-### 1. Inspect Pipelines
+## 📖 Usage
 
-List recent pipelines for a project:
+### Inspect Pipelines
 
 ```bash
 duoops pipelines list <project-id-or-path>
-# Example: duoops pipelines list my-group/my-project
 ```
 
-### 2. Analyze Jobs & Logs
-
-Get details and logs for a specific job:
+### Analyze Jobs & Logs
 
 ```bash
 duoops job show <project-id> <job-id>
-# Example: duoops job show 12345 987654
 ```
 
-### 3. Ask the AI Agent
-
-Troubleshoot failures or ask general CI questions:
+### Ask the AI Agent
 
 ```bash
 duoops ask "Why did the last pipeline fail?"
 duoops ask "How do I optimize my .gitlab-ci.yml for faster builds?"
 ```
 
-### 4. Measure Carbon Footprint
-
-Calculate emissions for a job using CPU/RAM usage data (JSON timeseries):
+### Measure Carbon Footprint
 
 ```bash
 duoops measure calculate \
@@ -108,54 +112,138 @@ duoops measure calculate \
   --out-md report.md
 ```
 
-This will output a report to the console and save a Markdown summary to `report.md`.
+### Get AI Recommendations on Emissions
 
-**Persist from CI:** To save results to BigQuery when the measure component runs in GitLab CI, set these CI/CD variables (masked where appropriate):
-
-| Variable | Description |
-| :--- | :--- |
-| `DUOOPS_BQ_DATASET` | BigQuery dataset name |
-| `DUOOPS_BQ_TABLE` | BigQuery table name |
-| `GCP_PROJECT_ID` | Google Cloud project ID |
+```bash
+duoops act [project-id] [--limit N]
+```
 
-The same GCP service account used for the measure component (e.g. `GCP_SA_KEY_BASE64`) is used for BigQuery; the component sets `GOOGLE_APPLICATION_CREDENTIALS` so the BigQuery client can authenticate. Ensure the table exists (e.g. run `duoops init` locally with BigQuery enabled to create it).
+Fetches recent carbon metrics from BigQuery and gets the agent's recommendations for reducing emissions.
 
-#### Act on recent data
+### Launch the Web Portal
 
-Run `duoops act [project-id] [--limit N]` to fetch recent CI carbon metrics from BigQuery and get the agent's recommendations (e.g. set a budget, optimize jobs, suggest .gitlab-ci changes). Project ID can be omitted if set via `duoops init`.
+```bash
+duoops portal
+```
 
-### 5. Launch the Web Portal
+Opens a local dashboard at `http://localhost:58327` for visualizing metrics and chatting with the AI assistant.
 
-Start the local dashboard to visualize metrics and chat:
+For a guided fix workflow:
 
 ```bash
-duoops portal
+duoops autofix
 ```
 
-Open your browser to `http://localhost:3000`.
+This launches the portal, switches to the console tab, and preloads a request asking the agent to inspect the latest pipeline failure.
+
+---
 
-## 🛠️ Development
+## 🔌 MCP Server
 
-### Project Structure
+The MCP server exposes DuoOps emissions data to AI agents via the [Model Context Protocol](https://modelcontextprotocol.io).
 
-*   `bin/`: CLI entry point.
-*   `src/commands/`: Oclif command implementations.
-*   `src/lib/`: Core logic (AI agents, GitLab client, Measurement tools).
-*   `portal/`: React frontend application.
+### Tools
 
-### Running Tests
+| Tool | Description |
+| :--- | :--- |
+| `get_job_emissions` | Retrieve emissions for a specific CI job |
+| `get_pipeline_emissions` | Aggregated emissions for a pipeline |
+| `get_project_emissions_summary` | Emissions trends for a project |
+| `get_machine_profile` | Specs and power data for cloud instances |
+| `get_region_carbon_intensity` | Grid carbon intensity for a region |
+
+### Deployment
 
 ```bash
-pnpm test
+# Using the CLI (recommended)
+duoops mcp deploy --gcp-project <PROJECT_ID> --region us-central1
+
+# Or manually with gcloud
+gcloud run deploy duoops-mcp \
+  --image registry.gitlab.com/youneslaaroussi/duoops/mcp-server:latest \
+  --set-env-vars "GCP_PROJECT_ID=<ID>,BQ_DATASET=<DATASET>,BQ_TABLE=<TABLE>" \
+  --allow-unauthenticated \
+  --region us-central1
 ```
 
-### Building for Production
+### GitLab Duo Chat Integration
+
+Add to your project's `.gitlab/duo/mcp.json`:
+
+```json
+{
+  "mcpServers": {
+    "duoops-carbon": {
+      "type": "http",
+      "url": "https://<your-cloud-run-url>/mcp",
+      "approvedTools": true
+    }
+  }
+}
+```
+
+---
+
+## 🔧 CI Component
+
+Include the measurement component in your `.gitlab-ci.yml`:
+
+```yaml
+include:
+  - component: gitlab.com/youneslaaroussi/duoops/duoops-measure-component@main
+    inputs:
+      provider: gcp
+      machine: e2-standard-4
+      region: us-central1
+```
+
+CI/CD variables for BigQuery persistence:
+
+| Variable | Description |
+| :--- | :--- |
+| `DUOOPS_BQ_DATASET` | BigQuery dataset name |
+| `DUOOPS_BQ_TABLE` | BigQuery table name |
+| `GCP_PROJECT_ID` | Google Cloud project ID |
+
+---
+
+## 🏗️ Architecture
+
+<p align="center">
+  <img src="hackathon-files/architecture.png" alt="DuoOps Architecture — Interfaces, AI Core, Data" width="700" />
+</p>
+
+---
+
+## 🛠️ Project Structure
+
+```
+duoops/
+├── bin/                  # CLI entry point
+├── src/                  # CLI commands and core logic
+│   ├── commands/         # Oclif command implementations
+│   └── lib/              # AI agents, GitLab client, measurement
+├── mcp-server/           # MCP server (standalone, deployable)
+├── portal/               # React web dashboard (Blueprint.js)
+├── templates/            # GitLab CI component templates
+├── data/                 # Sample data and fixtures
+└── test/                 # Test suite
+```
+
+## Development
 
 ```bash
+# Run tests
+pnpm test
+
+# Build everything
 pnpm build
+
+# Run the portal in dev mode
+cd portal && pnpm dev
 ```
 
-This compiles the TypeScript CLI and builds the React frontend, copying assets to the `dist/` directory.
+---
 
 ## 📄 License
 

package/data/aws_machine_power_profiles.json

@@ -0,0 +1,54 @@
+{
+  "m5.large": {
+    "vcpus": 2,
+    "memory_gb": 8,
+    "platform_cpu": "Intel Xeon Platinum 8175",
+    "matched_cpu_profile": "Intel Xeon Platinum 8175",
+    "cpu_power_profile": [
+      { "percentage": 0, "watts": 11.5 },
+      { "percentage": 10, "watts": 23.0 },
+      { "percentage": 50, "watts": 82.0 },
+      { "percentage": 100, "watts": 150.0 }
+    ],
+    "scope3_emissions_hourly": 18
+  },
+  "c5.xlarge": {
+    "vcpus": 4,
+    "memory_gb": 8,
+    "platform_cpu": "Intel Xeon Platinum 8275",
+    "matched_cpu_profile": "Intel Xeon Platinum 8275",
+    "cpu_power_profile": [
+      { "percentage": 0, "watts": 14.0 },
+      { "percentage": 10, "watts": 33.0 },
+      { "percentage": 50, "watts": 115.0 },
+      { "percentage": 100, "watts": 195.0 }
+    ],
+    "scope3_emissions_hourly": 22
+  },
+  "m6g.large": {
+    "vcpus": 2,
+    "memory_gb": 8,
+    "platform_cpu": "AWS Graviton2",
+    "matched_cpu_profile": "AWS Graviton2",
+    "cpu_power_profile": [
+      { "percentage": 0, "watts": 8.0 },
+      { "percentage": 10, "watts": 16.0 },
+      { "percentage": 50, "watts": 55.0 },
+      { "percentage": 100, "watts": 95.0 }
+    ],
+    "scope3_emissions_hourly": 15
+  },
+  "r6i.2xlarge": {
+    "vcpus": 8,
+    "memory_gb": 64,
+    "platform_cpu": "Intel Xeon Ice Lake",
+    "matched_cpu_profile": "Intel Xeon Ice Lake",
+    "cpu_power_profile": [
+      { "percentage": 0, "watts": 26.0 },
+      { "percentage": 10, "watts": 52.0 },
+      { "percentage": 50, "watts": 200.0 },
+      { "percentage": 100, "watts": 360.0 }
+    ],
+    "scope3_emissions_hourly": 42
+  }
+}

package/data/cpu_physical_specs.json

@@ -0,0 +1,105 @@
+{
+  "_metadata": {
+    "description": "Physical CPU specifications with vCPU counts for accurate power-per-vCPU calculations",
+    "last_updated": "2025-01-22",
+    "methodology": "Power per VM vCPU = (TDP / physical_vcpus) * vm_vcpus * utilization_ratio"
+  },
+  "Intel Xeon Gold 6268CL": {
+    "cores": 24,
+    "threads": 48,
+    "tdp_watts": 205,
+    "architecture": "Cascade Lake",
+    "source": "https://www.ebay.com/p/2321792675",
+    "source_name": "eBay Product Listing (SRF80)"
+  },
+  "Intel Xeon Gold 6253CL": {
+    "cores": 18,
+    "threads": 36,
+    "tdp_watts": 205,
+    "architecture": "Cascade Lake",
+    "source": "https://www.cpubenchmark.net/cpu.php?cpu=Intel+Xeon+Gold+6253CL+@+3.10GHz&id=4539",
+    "source_name": "PassMark CPU Benchmark"
+  },
+  "Intel Xeon Platinum 8481C": {
+    "cores": 56,
+    "threads": 112,
+    "tdp_watts": 350,
+    "architecture": "Sapphire Rapids",
+    "source": "https://www.techpowerup.com/cpu-specs/xeon-platinum-8481c.c3992",
+    "source_name": "TechPowerUp CPU Database"
+  },
+  "Intel Xeon Platinum 8373C": {
+    "cores": 36,
+    "threads": 72,
+    "tdp_watts": 300,
+    "architecture": "Ice Lake",
+    "source": "https://en.wikipedia.org/wiki/List_of_Intel_Xeon_processors_(Ice_Lake-based)",
+    "source_name": "Wikipedia - Intel Xeon Ice Lake"
+  },
+  "Intel Xeon Platinum 8280L": {
+    "cores": 28,
+    "threads": 56,
+    "tdp_watts": 205,
+    "architecture": "Cascade Lake",
+    "source": "https://ark.intel.com/content/www/us/en/ark/products/192472/intel-xeon-platinum-8280l-processor-38-5m-cache-2-70-ghz.html",
+    "source_name": "Intel ARK"
+  },
+  "Intel Xeon Platinum 8273CL": {
+    "cores": 28,
+    "threads": 56,
+    "tdp_watts": 165,
+    "architecture": "Cascade Lake",
+    "source": "https://en.wikipedia.org/wiki/List_of_Intel_Xeon_processors_(Cascade_Lake-based)",
+    "source_name": "Wikipedia - Intel Xeon Cascade Lake"
+  },
+  "Intel Xeon Scalable Platinum 8173M": {
+    "cores": 28,
+    "threads": 56,
+    "tdp_watts": 165,
+    "architecture": "Skylake",
+    "source": "https://en.wikipedia.org/wiki/List_of_Intel_Xeon_processors#702Skylake-based_703Xeon_Scalable",
+    "source_name": "Wikipedia - Intel Xeon Skylake"
+  },
+  "Intel Skylake": {
+    "cores": 28,
+    "threads": 56,
+    "tdp_watts": 150,
+    "architecture": "Skylake",
+    "source": "https://cloud.google.com/compute/docs/cpu-platforms",
+    "source_name": "Google Cloud CPU Platforms",
+    "note": "Generic Skylake profile for e2 instances, estimated from typical Skylake server CPUs"
+  },
+  "Intel Xeon E7-8880V4": {
+    "cores": 22,
+    "threads": 44,
+    "tdp_watts": 150,
+    "architecture": "Broadwell",
+    "source": "https://www.intel.com/content/www/us/en/products/sku/93792/intel-xeon-processor-e78880-v4-55m-cache-2-20-ghz/specifications.html",
+    "source_name": "Intel ARK"
+  },
+  "AMD EPYC 7B12": {
+    "cores": 64,
+    "threads": 128,
+    "tdp_watts": 240,
+    "architecture": "Rome (Zen 2)",
+    "source": "https://www.newegg.com/p/1FR-00G6-00026",
+    "source_name": "Newegg Product Listing"
+  },
+  "AMD EPYC 7B13": {
+    "cores": 64,
+    "threads": 128,
+    "tdp_watts": 225,
+    "architecture": "Milan (Zen 3)",
+    "source": "https://www.amd.com/en/products/cpu/amd-epyc-7b13",
+    "source_name": "AMD Product Page"
+  },
+  "Q64-30": {
+    "cores": 64,
+    "threads": 64,
+    "tdp_watts": 180,
+    "architecture": "ARM Neoverse N1",
+    "source": "https://amperecomputing.com/en/briefs/ampere-altra-family-product-brief",
+    "source_name": "Ampere Altra Family Product Brief",
+    "note": "ARM cores have no hyperthreading, threads = cores"
+  }
+}