cli-jaw 0.1.0

package/skills_ref/terraform/module-generation/skills/terraform-stacks/references/api-monitoring.md

@@ -0,0 +1,543 @@
+# API Monitoring Reference
+
+Complete guide for monitoring Terraform Stack deployments using the HCP Terraform API. Use this approach for automation, CI/CD pipelines, and non-interactive environments like AI agents.
+
+## Table of Contents
+
+1. [When to Use the API](#when-to-use-the-api)
+2. [Authentication](#authentication)
+3. [API Monitoring Workflow](#api-monitoring-workflow)
+4. [Detailed Endpoint Reference](#detailed-endpoint-reference)
+5. [Notes for AI Agents and Automation](#notes-for-ai-agents-and-automation)
+
+## When to Use the API
+
+Use the HCP Terraform API instead of CLI commands when:
+- Running in non-interactive environments (CI/CD, automation scripts)
+- Building tools or integrations that need programmatic access
+- Monitoring multiple Stacks simultaneously
+- Implementing custom retry logic or error handling
+- Working in environments where streaming CLI commands don't work
+
+**CLI commands that don't work in automation:**
+- `terraform stacks deployment-run watch` - Streams output, blocks indefinitely
+- `terraform stacks deployment-group watch` - Streams output, blocks indefinitely
+- `terraform stacks configuration watch` - Streams output, blocks indefinitely
+
+## Authentication
+
+### Extract API Token from Credentials File
+
+```bash
+TOKEN=$(jq -r '.credentials["app.terraform.io"].token' ~/.terraform.d/credentials.tfrc.json)
+```
+
+### Alternative: Use Environment Variable
+
+```bash
+export TFC_TOKEN="your-token-here"
+TOKEN=$TFC_TOKEN
+```
+
+### API Request Headers
+
+All API requests require these headers:
+
+```bash
+-H "Authorization: Bearer $TOKEN"
+-H "Content-Type: application/vnd.api+json"
+```
+
+## API Monitoring Workflow
+
+After uploading a configuration with `terraform stacks configuration upload`, follow this sequence to monitor deployment progress:
+
+### Step 1: Get Configuration Status
+
+**Endpoint:** `GET /api/v2/stack-configurations/{configuration-id}`
+
+**Purpose:** Verify configuration upload completed successfully and get the configuration details.
+
+**Request:**
+
+```bash
+curl -s -H "Authorization: Bearer $TOKEN" \
+  -H "Content-Type: application/vnd.api+json" \
+  "https://app.terraform.io/api/v2/stack-configurations/{configuration-id}" | jq '.'
+```
+
+**Response Fields:**
+- `attributes.status` - Configuration processing status (pending/completed)
+- `attributes.sequence-number` - Version number of this configuration
+- `attributes.components-detected` - Number of components found
+- `attributes.deployments-detected` - Number of deployments found
+
+**Example Response:**
+
+```json
+{
+  "data": {
+    "id": "stc-ABC123",
+    "type": "stack-configurations",
+    "attributes": {
+      "status": "completed",
+      "sequence-number": 5,
+      "components-detected": 3,
+      "deployments-detected": 2,
+      "created-at": "2024-01-15T10:30:00.000Z",
+      "updated-at": "2024-01-15T10:30:45.000Z"
+    }
+  }
+}
+```
+
+### Step 2: Get Deployment Group Summaries
+
+**Endpoint:** `GET /api/v2/stack-configurations/{configuration-id}/stack-deployment-group-summaries`
+
+**Purpose:** Get list of deployment groups, their IDs, and current status summary.
+
+**Request:**
+
+```bash
+curl -s -H "Authorization: Bearer $TOKEN" \
+  -H "Content-Type: application/vnd.api+json" \
+  "https://app.terraform.io/api/v2/stack-configurations/{configuration-id}/stack-deployment-group-summaries" | jq '.'
+```
+
+**Response Fields:**
+- `id` - Deployment group ID (needed for next step)
+- `attributes.name` - Deployment group name (e.g., `dev_default`)
+- `attributes.status` - Overall status (running/succeeded/failed)
+- `attributes.status-counts` - Breakdown of deployment statuses
+
+**Example Response:**
+
+```json
+{
+  "data": [
+    {
+      "id": "sdg-XYZ789",
+      "type": "stack-deployment-group-summaries",
+      "attributes": {
+        "name": "dev_default",
+        "status": "running",
+        "status-counts": {
+          "pending": 0,
+          "running": 1,
+          "succeeded": 1,
+          "failed": 0
+        }
+      }
+    }
+  ]
+}
+```
+
+### Step 3: Get Deployment Runs
+
+**Endpoint:** `GET /api/v2/stack-deployment-groups/{group-id}/stack-deployment-runs`
+
+**Purpose:** Get list of deployment runs for a specific group with their current status.
+
+**Request:**
+
+```bash
+curl -s -H "Authorization: Bearer $TOKEN" \
+  -H "Content-Type: application/vnd.api+json" \
+  "https://app.terraform.io/api/v2/stack-deployment-groups/{group-id}/stack-deployment-runs" | jq '.'
+```
+
+**Response Fields:**
+- `id` - Deployment run ID (needed for next step)
+- `attributes.status` - Current status (planning/planned/applying/applied/failed)
+- `attributes.created-at` - Run start time
+- `attributes.updated-at` - Last update time
+
+**Example Response:**
+
+```json
+{
+  "data": [
+    {
+      "id": "sdr-123ABC",
+      "type": "stack-deployment-runs",
+      "attributes": {
+        "status": "planning",
+        "created-at": "2024-01-15T10:31:00.000Z",
+        "updated-at": "2024-01-15T10:31:15.000Z"
+      }
+    }
+  ]
+}
+```
+
+### Step 4: Get Deployment Steps
+
+**Endpoint:** `GET /api/v2/stack-deployment-runs/{run-id}/stack-deployment-steps`
+
+**Purpose:** Get detailed information about individual plan and apply steps.
+
+**Request:**
+
+```bash
+curl -s -H "Authorization: Bearer $TOKEN" \
+  -H "Content-Type: application/vnd.api+json" \
+  "https://app.terraform.io/api/v2/stack-deployment-runs/{run-id}/stack-deployment-steps" | jq '.'
+```
+
+**Response Fields:**
+- `id` - Step ID (needed for diagnostics and outputs)
+- `attributes.operation-type` - Type of operation (plan/apply)
+- `attributes.status` - Step status (running/completed/failed)
+- `attributes.component-name` - Which component is being processed
+
+**Example Response:**
+
+```json
+{
+  "data": [
+    {
+      "id": "sds-PlanStep123",
+      "type": "stack-deployment-steps",
+      "attributes": {
+        "operation-type": "plan",
+        "status": "completed",
+        "component-name": "vpc",
+        "created-at": "2024-01-15T10:31:05.000Z",
+        "completed-at": "2024-01-15T10:31:30.000Z"
+      }
+    },
+    {
+      "id": "sds-ApplyStep456",
+      "type": "stack-deployment-steps",
+      "attributes": {
+        "operation-type": "apply",
+        "status": "running",
+        "component-name": "vpc",
+        "created-at": "2024-01-15T10:32:00.000Z"
+      }
+    }
+  ]
+}
+```
+
+### Step 5: Get Error Diagnostics (When Deployment Fails)
+
+**Endpoint:** `GET /api/v2/stack-deployment-steps/{step-id}/stack-diagnostics`
+
+**Purpose:** Retrieve detailed error messages when a deployment step fails.
+
+**Critical:** The `stack_deployment_step_id` query parameter is **required**. Without it, the API returns empty results.
+
+**Request:**
+
+```bash
+curl -s -H "Authorization: Bearer $TOKEN" \
+  -H "Content-Type: application/vnd.api+json" \
+  "https://app.terraform.io/api/v2/stack-deployment-steps/{step-id}/stack-diagnostics?stack_deployment_step_id={step-id}" | jq '.'
+```
+
+**Response Fields:**
+- `attributes.severity` - Diagnostic level (error/warning)
+- `attributes.summary` - Brief error description
+- `attributes.detail` - Detailed error message
+- `attributes.diags` - Array of diagnostic objects with file locations and code snippets
+
+**Example Response (Error with Details):**
+
+```json
+{
+  "data": [
+    {
+      "id": "stf-ErrorExampleId",
+      "type": "stack-diagnostics",
+      "attributes": {
+        "severity": "error",
+        "summary": "Diagnostics reported",
+        "detail": "2 errors",
+        "diags": [
+          {
+            "summary": "Unsupported attribute",
+            "detail": "This object does not have an attribute named \"target_id\".",
+            "range": {
+              "filename": "main.tf",
+              "start": {
+                "line": 634,
+                "column": 33
+              },
+              "end": {
+                "line": 634,
+                "column": 43
+              },
+              "source": "registry.terraform.io/terraform-aws-modules/alb/aws@9.17.0//main.tf"
+            },
+            "snippet": {
+              "code": "  target_id         = each.value.target_id",
+              "context": "resource \"aws_lb_target_group_attachment\" \"this\""
+            }
+          },
+          {
+            "summary": "Invalid reference",
+            "detail": "A reference to a resource type must be followed by at least one attribute access.",
+            "range": {
+              "filename": "main.tf",
+              "start": {
+                "line": 142,
+                "column": 15
+              },
+              "end": {
+                "line": 142,
+                "column": 28
+              },
+              "source": "local-module//main.tf"
+            },
+            "snippet": {
+              "code": "  vpc_id = aws_vpc.main",
+              "context": "resource \"aws_subnet\" \"private\""
+            }
+          }
+        ],
+        "acknowledged": false,
+        "created-at": "2024-01-15T10:32:15.000Z"
+      }
+    }
+  ]
+}
+```
+
+**Parsing Diagnostics:**
+
+Extract error information with jq:
+
+```bash
+# Get error summaries
+curl -s -H "Authorization: Bearer $TOKEN" \
+  "https://app.terraform.io/api/v2/stack-deployment-steps/{step-id}/stack-diagnostics?stack_deployment_step_id={step-id}" | \
+  jq -r '.data[].attributes.diags[]? | "\(.summary): \(.detail)"'
+
+# Get file locations
+curl -s -H "Authorization: Bearer $TOKEN" \
+  "https://app.terraform.io/api/v2/stack-deployment-steps/{step-id}/stack-diagnostics?stack_deployment_step_id={step-id}" | \
+  jq -r '.data[].attributes.diags[]? | "\(.range.filename):\(.range.start.line)"'
+```
+
+### Step 6: Get Stack Outputs (After Successful Deployment)
+
+**Endpoint:** `GET /api/v2/stack-deployment-steps/{final-apply-step-id}/artifacts?name=apply-description`
+
+**Purpose:** Retrieve Stack outputs after a successful deployment completes.
+
+**Important Notes:**
+- This endpoint returns HTTP 307 redirect - use `curl -L` to follow redirects automatically
+- This is currently the **only way** to retrieve Stack outputs programmatically
+- This endpoint is **not documented** in public API documentation
+- You need the final apply step ID from Step 4
+
+**Request:**
+
+```bash
+curl -L -s -H "Authorization: Bearer $TOKEN" \
+  "https://app.terraform.io/api/v2/stack-deployment-steps/{final-apply-step-id}/artifacts?name=apply-description"
+```
+
+**Response Structure:**
+
+The artifact response includes an `.outputs` object where each output contains a `change.after` property with the actual output value:
+
+```json
+{
+  "outputs": {
+    "alb_url": {
+      "change": {
+        "actions": ["no-op"],
+        "before": "http://my-alb-123456789.us-west-2.elb.amazonaws.com",
+        "after": "http://my-alb-123456789.us-west-2.elb.amazonaws.com",
+        "after_unknown": false,
+        "before_sensitive": false,
+        "after_sensitive": false
+      },
+      "type": "string"
+    },
+    "ecr_repository_url": {
+      "change": {
+        "actions": ["no-op"],
+        "before": "123456789.dkr.ecr.us-west-2.amazonaws.com/my-repo",
+        "after": "123456789.dkr.ecr.us-west-2.amazonaws.com/my-repo",
+        "after_unknown": false,
+        "before_sensitive": false,
+        "after_sensitive": false
+      },
+      "type": "string"
+    }
+  }
+}
+```
+
+**Extract Only Output Values:**
+
+```bash
+curl -L -s --header "Authorization: Bearer $TOKEN" \
+  "https://app.terraform.io/api/v2/stack-deployment-steps/{final-apply-step-id}/artifacts?name=apply-description" | \
+  jq -r '.outputs | to_entries | .[] | "\(.key): \(.value.change.after)"'
+```
+
+**Example Output:**
+
+```
+alb_url: http://my-alb-123456789.us-west-2.elb.amazonaws.com
+ecr_repository_url: 123456789.dkr.ecr.us-west-2.amazonaws.com/my-repo
+```
+
+## Detailed Endpoint Reference
+
+### Available Artifact Types
+
+The artifacts endpoint accepts these `name` parameter values:
+
+- `plan-description` - Terraform plan output in JSON format
+- `plan-debug-log` - Detailed debug logs from plan operation
+- `apply-description` - Terraform apply output including outputs (JSON format)
+- `apply-debug-log` - Detailed debug logs from apply operation
+
+### Polling Best Practices
+
+**Recommended polling intervals:**
+- Configuration status: Check every 5 seconds until status is "completed"
+- Deployment runs: Check every 10 seconds during active deployment
+- Deployment steps: Check every 10 seconds for individual step status
+
+**Implement exponential backoff:**
+
+```bash
+# Example polling script with backoff
+RETRY_COUNT=0
+MAX_RETRIES=30
+BACKOFF=5
+
+while [ $RETRY_COUNT -lt $MAX_RETRIES ]; do
+  STATUS=$(curl -s -H "Authorization: Bearer $TOKEN" \
+    "https://app.terraform.io/api/v2/stack-deployment-runs/{run-id}" | \
+    jq -r '.data.attributes.status')
+
+  if [ "$STATUS" = "applied" ] || [ "$STATUS" = "failed" ]; then
+    echo "Deployment finished with status: $STATUS"
+    break
+  fi
+
+  echo "Current status: $STATUS. Waiting ${BACKOFF}s..."
+  sleep $BACKOFF
+  RETRY_COUNT=$((RETRY_COUNT + 1))
+done
+```
+
+## Notes for AI Agents and Automation
+
+### CLI Command Limitations
+
+**These CLI commands DO NOT work in automation:**
+- `terraform stacks deployment-run watch` - Streams output, blocks indefinitely
+- `terraform stacks deployment-group watch` - Streams output, blocks indefinitely
+- `terraform stacks configuration watch` - Streams output, blocks indefinitely
+
+**Solution:** Use API polling instead of watch commands.
+
+### No Direct Output Command
+
+There is currently no CLI command to retrieve Stack outputs. You must:
+1. Use API to get deployment steps
+2. Find the final apply step ID
+3. Request the `apply-description` artifact
+4. Parse JSON to extract outputs
+
+### Handling Redirects
+
+The artifacts endpoint returns HTTP 307 redirect to the actual artifact location. Ensure your HTTP client follows redirects:
+
+**curl:** Use `-L` flag
+**Python requests:** Set `allow_redirects=True` (default)
+**Node.js fetch:** Set `redirect: 'follow'` (default)
+
+### Error Handling
+
+**Common API errors:**
+
+- **401 Unauthorized:** Invalid or expired token - refresh credentials
+- **404 Not Found:** Invalid ID or resource doesn't exist yet - retry with backoff
+- **429 Too Many Requests:** Rate limited - implement exponential backoff
+- **Empty diagnostics:** Missing required `stack_deployment_step_id` query parameter
+
+### Complete Monitoring Script Example
+
+```bash
+#!/bin/bash
+
+# Configuration
+TOKEN=$(jq -r '.credentials["app.terraform.io"].token' ~/.terraform.d/credentials.tfrc.json)
+CONFIG_ID="stc-ABC123"
+BASE_URL="https://app.terraform.io/api/v2"
+
+# Helper function
+api_get() {
+  curl -s -H "Authorization: Bearer $TOKEN" \
+    -H "Content-Type: application/vnd.api+json" \
+    "$1"
+}
+
+# 1. Wait for configuration to complete
+echo "Checking configuration status..."
+while true; do
+  STATUS=$(api_get "$BASE_URL/stack-configurations/$CONFIG_ID" | jq -r '.data.attributes.status')
+  [ "$STATUS" = "completed" ] && break
+  echo "Configuration status: $STATUS. Waiting..."
+  sleep 5
+done
+
+# 2. Get deployment groups
+echo "Getting deployment groups..."
+GROUP_ID=$(api_get "$BASE_URL/stack-configurations/$CONFIG_ID/stack-deployment-group-summaries" | \
+  jq -r '.data[0].id')
+
+# 3. Get deployment run
+echo "Getting deployment run..."
+RUN_ID=$(api_get "$BASE_URL/stack-deployment-groups/$GROUP_ID/stack-deployment-runs" | \
+  jq -r '.data[0].id')
+
+# 4. Monitor deployment run
+echo "Monitoring deployment run: $RUN_ID"
+while true; do
+  STATUS=$(api_get "$BASE_URL/stack-deployment-runs/$RUN_ID" | jq -r '.data.attributes.status')
+  echo "Deployment status: $STATUS"
+
+  if [ "$STATUS" = "applied" ]; then
+    echo "Deployment succeeded!"
+
+    # 5. Get outputs from final apply step
+    APPLY_STEP=$(api_get "$BASE_URL/stack-deployment-runs/$RUN_ID/stack-deployment-steps" | \
+      jq -r '.data[] | select(.attributes["operation-type"] == "apply") | .id' | tail -1)
+
+    echo "Retrieving outputs from step: $APPLY_STEP"
+    curl -L -s -H "Authorization: Bearer $TOKEN" \
+      "$BASE_URL/stack-deployment-steps/$APPLY_STEP/artifacts?name=apply-description" | \
+      jq -r '.outputs | to_entries | .[] | "\(.key): \(.value.change.after)"'
+    break
+  fi
+
+  if [ "$STATUS" = "failed" ]; then
+    echo "Deployment failed!"
+
+    # Get error diagnostics
+    FAILED_STEP=$(api_get "$BASE_URL/stack-deployment-runs/$RUN_ID/stack-deployment-steps" | \
+      jq -r '.data[] | select(.attributes.status == "failed") | .id' | head -1)
+
+    echo "Error diagnostics from step: $FAILED_STEP"
+    api_get "$BASE_URL/stack-deployment-steps/$FAILED_STEP/stack-diagnostics?stack_deployment_step_id=$FAILED_STEP" | \
+      jq -r '.data[].attributes.diags[]? | "\(.summary): \(.detail)"'
+    exit 1
+  fi
+
+  sleep 10
+done
+```
+
+This script demonstrates a complete monitoring workflow from configuration upload to output retrieval with error handling.