duoops 0.0.0

package/LICENSE

@@ -0,0 +1,22 @@
+MIT License
+
+Copyright (c) 2024 Younes Laaroussi
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+

package/README.md

@@ -0,0 +1,181 @@
+# DuoOps
+
+> **Toolset for Explainable and Sustainable CI on GitLab.**
+
+DuoOps is a developer-focused CLI and Web Portal designed to make GitLab CI/CD pipelines more transparent, debuggable, and environmentally sustainable. It leverages AI agents to explain failures and provides tools to measure and visualize the carbon footprint of your CI jobs.
+
+![License](https://img.shields.io/badge/license-MIT-blue.svg)
+![Version](https://img.shields.io/badge/version-0.0.0-green.svg)
+
+## 🌟 Key Features
+
+*   **🤖 AI-Powered Troubleshooting**: Ask questions about your pipelines in plain English. The AI agent (powered by GitLab Duo or Google Vertex AI) analyzes logs, identifies root causes, and suggests fixes.
+*   **🔍 Deep Pipeline Inspection**: View pipeline status, job details, and logs directly from your terminal.
+*   **🌱 Carbon Measurement**: Calculate the carbon emissions of your CI jobs based on CPU/RAM usage timeseries. Track your environmental impact over time.
+*   **📊 Web Portal**: A local React-based dashboard to visualize pipeline metrics and chat with the AI assistant in a rich interface.
+*   **🧠 Intelligent Summarization**: Automatically summarizes long job logs using sub-agents to provide concise failure analysis without token limit issues.
+
+## 🚀 Quick Start
+
+### Prerequisites
+
+*   **Node.js** >= 18.0.0
+*   **GitLab Account**: A Personal Access Token with `api` scope.
+*   **(Optional) Google Cloud SDK**: Required only for BigQuery metrics.
+    *   Install the `gcloud` CLI.
+    *   Authenticate: `gcloud auth application-default login`
+
+### Installation
+
+#### Install from npm (recommended)
+
+```bash
+# Install the published CLI globally
+npm install -g duoops
+
+# Verify the binary is available
+duoops --help
+```
+
+#### Install from source (development)
+
+```bash
+# Clone the repository
+git clone https://github.com/youneslaaroussi/duoops.git
+cd duoops
+
+# Install dependencies
+pnpm install
+
+# Build the project
+pnpm build
+
+# Link the CLI globally
+npm link
+```
+
+### Configuration
+
+Run the interactive initialization command to set up your credentials:
+
+```bash
+duoops init
+```
+
+This will prompt you for:
+*   GitLab URL (default: `https://gitlab.com`)
+*   GitLab Personal Access Token
+*   (Optional) BigQuery settings for sustainability tracking
+
+Alternatively, you can configure DuoOps using environment variables:
+
+| Variable | Description |
+| :--- | :--- |
+| `GITLAB_TOKEN` | Your GitLab Personal Access Token |
+| `GITLAB_URL` | GitLab instance URL (default: `https://gitlab.com`) |
+| `DUOOPS_AI_PROVIDER` | AI Provider: `gitlab` (default) or `vertex` |
+| `GCP_PROJECT_ID` | Google Cloud Project ID (required for Vertex AI) |
+| `GCP_LOCATION` | Google Cloud Location (default: `us-central1`) |
+
+## 📖 Usage
+
+### 1. Inspect Pipelines
+
+List recent pipelines for a project:
+
+```bash
+duoops pipelines list <project-id-or-path>
+# Example: duoops pipelines list my-group/my-project
+```
+
+### 2. Analyze Jobs & Logs
+
+Get details and logs for a specific job:
+
+```bash
+duoops job show <project-id> <job-id>
+# Example: duoops job show 12345 987654
+```
+
+### 3. Ask the AI Agent
+
+Troubleshoot failures or ask general CI questions:
+
+```bash
+duoops ask "Why did the last pipeline fail?"
+duoops ask "How do I optimize my .gitlab-ci.yml for faster builds?"
+```
+
+### 4. Measure Carbon Footprint
+
+Calculate emissions for a job using CPU/RAM usage data (JSON timeseries):
+
+```bash
+duoops measure calculate \
+  --provider gcp \
+  --machine e2-standard-4 \
+  --region us-central1 \
+  --cpu-timeseries ./data/cpu.json \
+  --ram-used-timeseries ./data/ram.json \
+  --out-md report.md
+```
+
+This will output a report to the console and save a Markdown summary to `report.md`.
+
+**Persist from CI:** To save results to BigQuery when the measure component runs in GitLab CI, set these CI/CD variables (masked where appropriate):
+
+| Variable | Description |
+| :--- | :--- |
+| `DUOOPS_BQ_DATASET` | BigQuery dataset name |
+| `DUOOPS_BQ_TABLE` | BigQuery table name |
+| `GCP_PROJECT_ID` | Google Cloud project ID |
+
+The same GCP service account used for the measure component (e.g. `GCP_SA_KEY_BASE64`) is used for BigQuery; the component sets `GOOGLE_APPLICATION_CREDENTIALS` so the BigQuery client can authenticate. Ensure the table exists (e.g. run `duoops init` locally with BigQuery enabled to create it).
+
+#### Act on recent data
+
+Run `duoops act [project-id] [--limit N]` to fetch recent CI carbon metrics from BigQuery and get the agent's recommendations (e.g. set a budget, optimize jobs, suggest .gitlab-ci changes). Project ID can be omitted if set via `duoops init`.
+
+### 5. Launch the Web Portal
+
+Start the local dashboard to visualize metrics and chat:
+
+```bash
+duoops portal
+```
+
+Open your browser to `http://localhost:3000`.
+
+## 🛠️ Development
+
+### Project Structure
+
+*   `bin/`: CLI entry point.
+*   `src/commands/`: Oclif command implementations.
+*   `src/lib/`: Core logic (AI agents, GitLab client, Measurement tools).
+*   `portal/`: React frontend application.
+
+### Running Tests
+
+```bash
+pnpm test
+```
+
+### Building for Production
+
+```bash
+pnpm build
+```
+
+This compiles the TypeScript CLI and builds the React frontend, copying assets to the `dist/` directory.
+
+## 📦 Publishing to npm
+
+1.  Update `package.json` with the release version you want to publish.
+2.  Run the checks: `pnpm test` (this runs lint afterwards) to ensure the CLI and portal build cleanly.
+3.  Inspect the publish payload locally with `pnpm pack` (this runs the `prepack` script, which now builds the CLI, portal, and Oclif manifest automatically). Untar the generated `.tgz` if you want to double-check the contents.
+4.  When you're satisfied, publish the package: `pnpm publish --access public`.
+
+## 📄 License
+
+MIT

package/bin/dev.cmd

@@ -0,0 +1,3 @@
+@echo off
+
+node --loader ts-node/esm --no-warnings=ExperimentalWarning "%~dp0\dev" %*

package/bin/dev.js

@@ -0,0 +1,7 @@
+#!/usr/bin/env -S node --loader ts-node/esm --disable-warning=ExperimentalWarning
+
+// eslint-disable-next-line n/no-unpublished-import
+import 'dotenv/config'
+import {execute} from '@oclif/core'
+
+await execute({development: true, dir: import.meta.url})

package/bin/run.cmd

@@ -0,0 +1,3 @@
+@echo off
+
+node "%~dp0\run" %*

package/bin/run.js

@@ -0,0 +1,8 @@
+#!/usr/bin/env node
+
+// Suppress AI SDK compatibility warnings (e.g. specificationVersion v2) when portal runs
+globalThis.AI_SDK_LOG_WARNINGS = false
+
+import {execute} from '@oclif/core'
+
+await execute({dir: import.meta.url})

package/dist/commands/act.d.ts

@@ -0,0 +1,12 @@
+import { Command } from '@oclif/core';
+export default class Act extends Command {
+    static args: {
+        'project-id': import("@oclif/core/interfaces").Arg<string | undefined, Record<string, unknown>>;
+    };
+    static description: string;
+    static examples: string[];
+    static flags: {
+        limit: import("@oclif/core/interfaces").OptionFlag<number, import("@oclif/core/interfaces").CustomOptions>;
+    };
+    run(): Promise<void>;
+}

package/dist/commands/act.js

@@ -0,0 +1,61 @@
+import { Args, Command, Flags } from '@oclif/core';
+import { runAgent } from '../lib/ai/agent.js';
+import { configManager } from '../lib/config.js';
+import { fetchCarbonMetrics } from '../lib/measure/bigquery-service.js';
+export default class Act extends Command {
+    static args = {
+        'project-id': Args.string({
+            description: 'GitLab project ID (or use default from duoops init)',
+            required: false,
+        }),
+    };
+    static description = 'Fetch recent CI carbon metrics from BigQuery and get the agent\'s recommendations';
+    static examples = [
+        `<%= config.bin %> <%= command.id %>`,
+        `<%= config.bin %> <%= command.id %> 12345 --limit 20`,
+    ];
+    static flags = {
+        limit: Flags.integer({
+            char: 'n',
+            default: 20,
+            description: 'Number of recent job records to fetch',
+        }),
+    };
+    async run() {
+        const { args, flags } = await this.parse(Act);
+        const config = configManager.get();
+        const projectId = args['project-id'] ?? config.defaultProjectId;
+        if (!projectId) {
+            this.error('Project ID is required. Pass it as an argument or run "duoops init" and set a default project.');
+        }
+        this.log('Fetching recent carbon metrics...');
+        let metrics;
+        try {
+            metrics = await fetchCarbonMetrics(projectId, flags.limit);
+        }
+        catch (error) {
+            const message = error instanceof Error ? error.message : String(error);
+            this.error(message.includes('BigQuery') ? message : `Failed to fetch metrics: ${message}`);
+        }
+        if (metrics.length === 0) {
+            this.log('No carbon metrics found for this project. Run the measure component in CI with BigQuery env vars set to persist data.');
+            return;
+        }
+        const dataBlob = JSON.stringify(metrics, null, 2);
+        const prompt = `Here are the last ${metrics.length} CI job carbon metrics for GitLab project ${projectId} (from BigQuery):
+
+\`\`\`json
+${dataBlob}
+\`\`\`
+
+Analyze this data and recommend concrete actions. For example: set an emission budget, add or tune the measure component in .gitlab-ci.yml, optimize the heaviest jobs, suggest region or machine type changes, or link to specific job logs. Be concise and actionable.`;
+        this.log('Thinking...');
+        try {
+            const response = await runAgent(prompt);
+            this.log('\n' + response);
+        }
+        catch (error) {
+            this.error(error instanceof Error ? error.message : String(error));
+        }
+    }
+}

package/dist/commands/ask.d.ts

@@ -0,0 +1,8 @@
+import { Command } from '@oclif/core';
+export default class Ask extends Command {
+    static args: {
+        question: import("@oclif/core/interfaces").Arg<string, Record<string, unknown>>;
+    };
+    static description: string;
+    run(): Promise<void>;
+}

package/dist/commands/ask.js

@@ -0,0 +1,22 @@
+import { Args, Command } from '@oclif/core';
+import { runAgent } from '../lib/ai/agent.js';
+export default class Ask extends Command {
+    static args = {
+        question: Args.string({
+            description: 'The question to ask the agent',
+            required: true,
+        }),
+    };
+    static description = 'Ask questions about your CI/CD pipelines, logs, and sustainability';
+    async run() {
+        const { args } = await this.parse(Ask);
+        this.log('Thinking...');
+        try {
+            const response = await runAgent(args.question);
+            this.log('\n' + response);
+        }
+        catch (error) {
+            this.error(error instanceof Error ? error.message : String(error));
+        }
+    }
+}

package/dist/commands/init.d.ts

@@ -0,0 +1,5 @@
+import { Command } from '@oclif/core';
+export default class Init extends Command {
+    static description: string;
+    run(): Promise<void>;
+}

package/dist/commands/init.js

@@ -0,0 +1,97 @@
+import { BigQuery } from '@google-cloud/bigquery';
+import { confirm, input, password } from '@inquirer/prompts';
+import { Command } from '@oclif/core';
+import { configManager } from '../lib/config.js';
+export default class Init extends Command {
+    static description = 'Initialize the configuration for DuoOps';
+    async run() {
+        this.log('Welcome to DuoOps! Let\'s set up your configuration.');
+        this.log('You will need a GitLab Personal Access Token with the "api" scope.');
+        this.log('You can generate one at: https://gitlab.com/-/user_settings/personal_access_tokens');
+        this.log('');
+        const gitlabUrl = await input({
+            default: 'https://gitlab.com',
+            message: 'GitLab URL',
+        });
+        const gitlabToken = await password({
+            mask: '*',
+            message: 'GitLab Personal Access Token',
+        });
+        const enableMeasure = await confirm({
+            message: 'Enable Measure/Sustainability Tracking (BigQuery)?',
+        });
+        let bigqueryDataset;
+        let bigqueryTable;
+        let googleProjectId;
+        if (enableMeasure) {
+            this.log('Note: You must have Google Cloud SDK installed and authenticated (gcloud auth application-default login) or a service account key.');
+            googleProjectId = await input({
+                message: 'Google Cloud Project ID',
+            });
+            bigqueryDataset = await input({
+                default: 'measure_data',
+                message: 'BigQuery Dataset Name',
+            });
+            bigqueryTable = await input({
+                default: 'emissions',
+                message: 'BigQuery Table Name',
+            });
+            const shouldCreate = await confirm({
+                message: `Attempt to create BigQuery Dataset (${bigqueryDataset}) and Table (${bigqueryTable}) if missing?`,
+            });
+            if (shouldCreate) {
+                try {
+                    this.log('Checking BigQuery resources...');
+                    const bigquery = new BigQuery({ projectId: googleProjectId });
+                    // 1. Create Dataset
+                    const dataset = bigquery.dataset(bigqueryDataset);
+                    const [datasetExists] = await dataset.exists();
+                    if (datasetExists) {
+                        this.log(`Dataset '${bigqueryDataset}' already exists.`);
+                    }
+                    else {
+                        this.log(`Creating dataset '${bigqueryDataset}'...`);
+                        await dataset.create({ location: 'US' }); // Default location
+                        this.log('Dataset created.');
+                    }
+                    // 2. Create Table with Schema
+                    const table = dataset.table(bigqueryTable);
+                    const [tableExists] = await table.exists();
+                    if (tableExists) {
+                        this.log(`Table '${bigqueryTable}' already exists.`);
+                    }
+                    else {
+                        this.log(`Creating table '${bigqueryTable}'...`);
+                        const schema = [
+                            { name: 'ingested_at', type: 'TIMESTAMP' },
+                            { name: 'gitlab_project_id', type: 'INTEGER' },
+                            { name: 'gitlab_job_id', type: 'INTEGER' },
+                            { name: 'gitlab_job_name', type: 'STRING' },
+                            { name: 'gitlab_user_name', type: 'STRING' },
+                            { name: 'machine_type', type: 'STRING' },
+                            { name: 'region', type: 'STRING' },
+                            { name: 'runtime_seconds', type: 'FLOAT' },
+                            { name: 'cpu_utilization_avg', type: 'FLOAT' },
+                            { name: 'ram_utilization_avg', type: 'FLOAT' },
+                            { name: 'energy_kwh', type: 'FLOAT' },
+                            { name: 'total_emissions_g', type: 'FLOAT' }
+                        ];
+                        await table.create({ schema });
+                        this.log('Table created successfully.');
+                    }
+                }
+                catch (error) {
+                    this.warn(`Failed to create BigQuery resources: ${error instanceof Error ? error.message : String(error)}`);
+                    this.warn('You may need to create them manually or check your permissions.');
+                }
+            }
+        }
+        configManager.set({
+            gitlabToken,
+            gitlabUrl,
+            measure: enableMeasure ? { bigqueryDataset, bigqueryTable, googleProjectId } : undefined,
+        });
+        this.log('Configuration saved successfully!');
+        this.log(`You can now use DuoOps commands. Try running '${this.config.bin} pipelines:list <project-id>'`);
+    }
+}

package/dist/commands/job/logs.d.ts

@@ -0,0 +1,13 @@
+import { Command } from '@oclif/core';
+export default class JobLogs extends Command {
+    static args: {
+        project: import("@oclif/core/interfaces").Arg<string, Record<string, unknown>>;
+        job_id: import("@oclif/core/interfaces").Arg<number, {
+            max?: number;
+            min?: number;
+        }>;
+    };
+    static description: string;
+    static examples: string[];
+    run(): Promise<void>;
+}

package/dist/commands/job/logs.js

@@ -0,0 +1,26 @@
+import { Args, Command } from '@oclif/core';
+import { getPipelineProvider } from '../../lib/gitlab/index.js';
+export default class JobLogs extends Command {
+    static args = {
+        /* eslint-disable perfectionist/sort-objects -- project must be first for CLI arg order */
+        project: Args.string({
+            description: 'Project ID or path (e.g. group/project)',
+            required: true,
+        }),
+        job_id: Args.integer({
+            description: 'Job ID',
+            required: true,
+        }),
+        /* eslint-enable perfectionist/sort-objects */
+    };
+    static description = 'Get job log trace';
+    static examples = [
+        `<%= config.bin %> <%= command.id %> group/my-project 12345`,
+    ];
+    async run() {
+        const { args } = await this.parse(JobLogs);
+        const provider = getPipelineProvider();
+        const trace = await provider.getJobTrace(args.project, args.job_id);
+        this.log(trace);
+    }
+}

package/dist/commands/measure/calculate.d.ts

@@ -0,0 +1,19 @@
+import { Command } from '@oclif/core';
+export default class CarbonCalculate extends Command {
+    static args: {};
+    static description: string;
+    static examples: string[];
+    static flags: {
+        budget: import("@oclif/core/interfaces").OptionFlag<number | undefined, import("@oclif/core/interfaces").CustomOptions>;
+        'cpu-timeseries': import("@oclif/core/interfaces").OptionFlag<string, import("@oclif/core/interfaces").CustomOptions>;
+        'fail-on-budget': import("@oclif/core/interfaces").BooleanFlag<boolean>;
+        machine: import("@oclif/core/interfaces").OptionFlag<string, import("@oclif/core/interfaces").CustomOptions>;
+        'out-json': import("@oclif/core/interfaces").OptionFlag<string | undefined, import("@oclif/core/interfaces").CustomOptions>;
+        'out-md': import("@oclif/core/interfaces").OptionFlag<string | undefined, import("@oclif/core/interfaces").CustomOptions>;
+        provider: import("@oclif/core/interfaces").OptionFlag<string, import("@oclif/core/interfaces").CustomOptions>;
+        'ram-size-timeseries': import("@oclif/core/interfaces").OptionFlag<string | undefined, import("@oclif/core/interfaces").CustomOptions>;
+        'ram-used-timeseries': import("@oclif/core/interfaces").OptionFlag<string, import("@oclif/core/interfaces").CustomOptions>;
+        region: import("@oclif/core/interfaces").OptionFlag<string, import("@oclif/core/interfaces").CustomOptions>;
+    };
+    run(): Promise<void>;
+}