@quiltdata/benchling-webhook 0.4.13 → 0.5.0

package/AGENTS.md

@@ -1,226 +0,0 @@
-# Benchling Webhook Integration - Complete Guide
-
-Complete deployment and operational guide for the Benchling webhook integration with Quilt.
-
-## Architecture Overview
-
-This AWS CDK application deploys a highly available, auto-scaling webhook processor using:
-
-- **Amazon API Gateway** → Routes HTTPS webhooks with IP-based access control
-- **Application Load Balancer (ALB)** → Distributes traffic across container instances
-- **AWS Fargate on Amazon ECS** → Runs containerized webhook processor (auto-scales 2-10 tasks)
-- **Amazon S3** → Stores webhook payloads and package data
-- **Amazon SQS** → Queues package creation requests for Quilt
-- **AWS Secrets Manager** → Securely stores Benchling OAuth credentials
-- **Amazon CloudWatch** → Provides centralized logging and monitoring
-- **AWS IAM** → Enforces least-privilege access controls
-
-**Request Flow:** Benchling → API Gateway → ALB → Fargate (Flask app) → S3 + SQS
-
-### Code Organization
-
-- **Infrastructure (CDK)**: `bin/` and `lib/` contain TypeScript CDK code for AWS deployment
-  - `lib/benchling-webhook-stack.ts` - Main stack orchestrating all components
-  - `lib/fargate-service.ts` - ECS Fargate service running Flask in Docker
-  - `lib/alb-api-gateway.ts` - API Gateway with HTTP integration to ALB
-  - `lib/ecr-repository.ts` - Docker image repository
-- **Application (Python)**: `docker/` contains Flask webhook processor
-  - See [docker/README.md](docker/README.md) for application development
-
-## Prerequisites
-
-- **AWS Account** with appropriate IAM permissions
-- **AWS CLI** v2.x configured with credentials
-- **Node.js** >= 18.0.0
-- **Docker** for container builds
-- **Quilt Stack** deployed with S3 bucket and SQS queue configured
-- **Benchling Account** with app creation permissions
-
-## Installation
-
-### 1. Clone and Install
-
-```bash
-git clone https://github.com/quiltdata/benchling-webhook.git
-cd benchling-webhook
-npm install
-```
-
-### 2. Configure Environment
-
-#### Option A: Auto-infer from Quilt Catalog (Recommended)
-
-If you have an existing Quilt deployment, you can automatically infer most configuration values:
-
-```bash
-# Infer config from your Quilt catalog
-npm run get-env -- https://quilt-catalog.yourcompany.com --write
-
-# Review the generated env.inferred file
-cat env.inferred
-
-# Copy to .env and fill in Benchling credentials
-cp env.inferred .env
-# Then edit .env to add your Benchling-specific values
-```
-
-The script will:
-
-- Fetch `config.json` from your Quilt catalog
-- Query AWS CloudFormation to find your Quilt stack
-- Extract bucket names, queue names, region, and account ID
-- Generate a `.env.inferred` file with pre-filled AWS/Quilt configuration
-
-**Note:** You'll still need to manually add Benchling credentials (tenant, client ID, client secret, etc.).
-
-#### Option B: Manual Configuration
-
-```bash
-cp env.template .env
-```
-
-Edit `.env` with your configuration:
-
-**Required Variables** (you must provide these):
-
-| Variable | Description |
-|----------|-------------|
-| `QUILT_CATALOG` | Quilt catalog URL (e.g., `quilt-catalog.yourcompany.com`) |
-| `QUILT_USER_BUCKET` | Your S3 bucket for Benchling exports |
-| `BENCHLING_TENANT` | Benchling subdomain (e.g., `myorg` from `myorg.benchling.com`) |
-| `BENCHLING_CLIENT_ID` | OAuth client ID from Benchling app |
-| `BENCHLING_CLIENT_SECRET` | OAuth client secret from Benchling app |
-| `BENCHLING_APP_DEFINITION_ID` | App definition ID for webhook verification |
-
-**Auto-Inferred Variables** (automatically determined from your Quilt catalog):
-
-| Variable | How It's Inferred |
-|----------|-------------------|
-| `CDK_DEFAULT_ACCOUNT` | From AWS STS (your current account) |
-| `CDK_DEFAULT_REGION` | From catalog config.json |
-| `QUEUE_NAME` | From Quilt stack outputs |
-| `SQS_QUEUE_URL` | From Quilt stack outputs |
-| `QUILT_DATABASE` | From Quilt stack outputs |
-
-**Optional Variables** (have sensible defaults):
-
-| Variable | Default | Description |
-|----------|---------|-------------|
-| `PKG_PREFIX` | `benchling` | Quilt package name prefix |
-| `PKG_KEY` | `experiment_id` | Metadata key for linking entries to packages |
-| `LOG_LEVEL` | `INFO` | Logging level (DEBUG, INFO, WARNING, ERROR, CRITICAL) |
-| `ENABLE_WEBHOOK_VERIFICATION` | `true` | Verify webhook signatures |
-| `WEBHOOK_ALLOW_LIST` | (empty) | Comma-separated IP allowlist |
-| `ECR_REPOSITORY_NAME` | `quiltdata/benchling` | Custom ECR repo name |
-
-See [doc/PARAMETERS.md](doc/PARAMETERS.md) for complete reference.
-
-### 3. Deploy Infrastructure
-
-```bash
-# Bootstrap CDK (first time only)
-source .env
-npx cdk bootstrap aws://$CDK_DEFAULT_ACCOUNT/$CDK_DEFAULT_REGION
-
-# Deploy stack
-npm run deploy
-```
-
-The webhook URL will be saved to `.env.deploy`:
-
-```bash
-WEBHOOK_ENDPOINT=https://abc123.execute-api.us-east-1.amazonaws.com/prod
-```
-
-## Post-Deployment Configuration
-
-### Configure Benchling App
-
-1. **Create App**: Benchling → Developer Console → Apps → Create app → From manifest
-2. **Upload Manifest**: Use `app-manifest.yaml` from this repository
-3. **Set Credentials**: Create Client Secret → Copy ID and Secret to `.env`
-4. **Configure Webhook**: Overview → Webhook URL → Paste URL from `.env.deploy`
-5. **Install App**: Version History → Install → Activate
-6. **Grant Permissions**: Tenant Admin → Organizations → Apps → Add app → Set role to Admin
-
-### Verify Deployment
-
-```bash
-# Health check
-source .env.deploy
-curl $WEBHOOK_ENDPOINT/health
-
-# Monitor logs
-aws logs tail /ecs/benchling-webhook --follow
-```
-
-## Usage
-
-1. **Create Entry** in Benchling notebook
-2. **Insert Canvas** → Select "Quilt Integration"
-3. **Create Package** → Generates versioned Quilt package
-4. **Add Files** → Attach experimental data
-5. **Update Package** → Creates new version with attachments
-
-## Development
-
-### Common Commands
-
-**Development:**
-- `npm run build` - Compile TypeScript
-- `npm run test` - Run Jest tests
-- `npm run lint` - Apply ESLint
-
-**Deployment:**
-- `npm run deploy` - Test + deploy (outputs to `.env.deploy`)
-- `npm run docker-push` - Build and push Docker images
-- `npm run docker-check` - Validate Docker images
-- `npm run release` - Create production release
-
-**Python App:**
-- See [docker/README.md](docker/README.md) or run `make help` in docker/ directory
-
-### Coding Style
-
-- **TypeScript**: 4-space indent, double quotes, trailing commas, required semicolons
-- **Types**: Avoid `any` in production; explicit return types on exports
-- **Organization**: Separate CDK constructs in `lib/`; application code in `docker/`
-
-### Commits & PRs
-
-- Use Conventional Commits: `type(scope): summary`
-- Keep commits focused; update `package-lock.json` when needed
-- Include test results and deployment considerations in PRs
-
-## Security Best Practices
-
-- OAuth credentials stored in AWS Secrets Manager
-- IP-based access control via API Gateway resource policies
-- Container images scanned for vulnerabilities via Amazon ECR
-- IAM roles follow least-privilege principle
-- All traffic encrypted in transit (TLS 1.2+)
-- CloudWatch logs encrypted at rest
-
-## Monitoring & Troubleshooting
-
-### Monitoring
-
-- **CloudWatch Logs**: `/ecs/benchling-webhook`
-- **ECS Task Metrics**: CPU, memory, task count
-- **API Gateway Metrics**: Request count, latency, 4XX/5XX errors
-- **ALB Target Health**: Monitor unhealthy targets
-
-### Health Endpoints
-
-- `/health` - General health check
-- `/health/ready` - Readiness probe
-
-### Debugging
-
-- **Deployment**: Check `.env.deploy` for outputs
-- **Logs**: `npm run logs` or `aws logs tail /ecs/benchling-webhook --follow`
-- **Events**: `npm run event` to send test events
-
-## License
-
-Apache-2.0 - See [LICENSE](LICENSE) file for details

package/CHANGELOG.md

@@ -1,91 +0,0 @@
-<!-- markdownlint-disable MD024 -->
-# Changelog
-
-All notable changes to this project will be documented in this file.
-
-## [Unreleased]
-
-## [0.4.13] - 2025-10-28
-
-### Changed
-
-- **Simplified configuration**: Auto-infer AWS and Quilt config from catalog at deployment time
-- **Reduced required .env variables**: From 20+ to just 6 (catalog URL, user bucket, 4 Benchling credentials)
-- **Renamed `BUCKET_NAME` → `QUILT_USER_BUCKET`**: Clearer distinction between user data bucket and Quilt system buckets
-- **Added CDK bootstrap validation**: Fails fast with helpful error if account/region not bootstrapped
-- **Made LOG_LEVEL configurable**: Override default INFO level for production debugging
-
-### Removed
-
-- Unused environment variables: `BENCHLING_API_KEY`, `PKG_BUCKET_ONLY`, `PREFIX`, `STAGE`, `FLASK_ENV`
-
-### Fixed
-
-- Module import issue: CLI argument parsing now only runs when executed directly, not on import
-
-## [0.4.12] - 2025-10-27
-
-### Added
-
-- Dev release workflow with timestamped pre-release tags for testing CI/CD pipeline
-
-### Changed
-
-- Refactored release script to separate version bumping from tag creation
-- version.js now outputs just the version number when called with no arguments
-
-## [0.4.11] - 2025-10-27
-
-### Added
-
-- Version synchronization test to ensure package.json, docker/pyproject.toml, and docker/app-manifest.yaml remain in sync
-- app-manifest.yaml now published as GitHub release asset for Benchling App installations
-
-### Fixed
-
-- Version bump script (bin/version.js) now updates all three version files instead of just package.json
-- `docker-validate` target now validates ECR repository is publicly accessible without authentication
-- `docker-validate` reads Docker image URI from `cdk-outputs.json` instead of requiring version parameter
-- `docker-validate` will fail if repository requires authentication, ensuring public access is maintained
-
-## [0.4.10] - 2025-10-27
-
-### Added
-
-- Canvas error notification section to display warnings and errors to users
-- Athena permissions (StartQueryExecution, GetQueryExecution, GetQueryResults) to ECS task role
-- Glue Data Catalog permissions for Athena queries
-- S3 permissions for Athena query results bucket
-- Test event file for Athena access denied scenario
-
-### Fixed
-
-- Canvas now displays error notifications instead of failing silently when PackageQuery encounters AWS permission issues
-- Improved error messages for Athena AccessDeniedException with actionable guidance
-
-## [0.4.9] - 2025-10-27
-
-### Added
-
-- Integrated release workflow into CI pipeline for automated GitHub releases, Docker image publishing, and NPM package publishing
-- Support for both production and pre-release (dev) versions
-
-### Changed
-
-- Updated Python to 3.14 in CI workflows
-- Updated aws-actions/configure-aws-credentials to v5
-- Updated actions/setup-python to v6
-- Streamlined release process with automated tagging and publishing
-
-## [0.4.8] - 2025-10-27
-
-### Changed
-
-- **Infrastructure Migration** - Migrated from Lambda to Docker/Fargate for improved scalability and resource management
-- **Improved Deployment** - Streamlined Docker-based deployment workflow with health checks and automated verification
-- **Enhanced Testing** - Added comprehensive test commands for local development and CI/CD workflows
-
-### Fixed
-
-- Resolved CloudFormation deployment conflicts during stack updates
-- Ensured ECR repository exists before Docker push in CI

package/bin/benchling-webhook.ts

@@ -1,172 +0,0 @@
-#!/usr/bin/env node
-import "dotenv/config";
-import * as cdk from "aws-cdk-lib";
-import { BenchlingWebhookStack } from "../lib/benchling-webhook-stack";
-
-// Import get-env to infer configuration from catalog
-const { inferStackConfig } = require("./get-env.js");
-const { execSync } = require("child_process");
-
-/**
- * Check if CDK is bootstrapped for the given account/region
- */
-async function checkCdkBootstrap(account: string, region: string): Promise<void> {
-    try {
-        console.log(`Checking CDK bootstrap for account ${account} in ${region}...`);
-
-        // Check if bootstrap stack exists
-        const result = execSync(
-            `aws cloudformation describe-stacks --region ${region} --stack-name CDKToolkit --query "Stacks[0].StackStatus" --output text 2>&1`,
-            { encoding: "utf-8" }
-        );
-
-        const stackStatus = result.trim();
-
-        if (stackStatus.includes("does not exist") || stackStatus.includes("ValidationError")) {
-            console.error("\n❌ CDK Bootstrap Error");
-            console.error("=".repeat(80));
-            console.error(`CDK is not bootstrapped for account ${account} in region ${region}`);
-            console.error("\nTo bootstrap CDK, run:");
-            console.error(`  npx cdk bootstrap aws://${account}/${region}`);
-            console.error("\nOr source your .env and run:");
-            console.error(`  source .env`);
-            console.error(`  npx cdk bootstrap aws://\${CDK_DEFAULT_ACCOUNT}/\${CDK_DEFAULT_REGION}`);
-            console.error("=".repeat(80));
-            process.exit(1);
-        }
-
-        // Check if the stack is in a good state
-        if (!stackStatus.includes("COMPLETE")) {
-            console.error("\n⚠️  CDK Bootstrap Warning");
-            console.error("=".repeat(80));
-            console.error(`CDKToolkit stack is in state: ${stackStatus}`);
-            console.error("This may cause deployment issues.");
-            console.error("=".repeat(80));
-        } else {
-            console.log(`✓ CDK is bootstrapped (CDKToolkit stack: ${stackStatus})\n`);
-        }
-    } catch (error) {
-        console.error("\n⚠️  Warning: Could not verify CDK bootstrap status");
-        console.error(`Error: ${(error as Error).message}`);
-        console.error("\nProceeding anyway, but deployment may fail if CDK is not bootstrapped.\n");
-    }
-}
-
-/**
- * Get environment configuration with catalog inference
- *
- * This combines user-provided values from .env with inferred values from the Quilt catalog.
- * User values always take precedence over inferred values.
- */
-async function getConfig() {
-    const userEnv = process.env;
-    let inferredEnv: Record<string, string> = {};
-
-    // If QUILT_CATALOG is provided, try to infer additional configuration
-    if (userEnv.QUILT_CATALOG) {
-        try {
-            console.log(`Inferring configuration from catalog: ${userEnv.QUILT_CATALOG}`);
-            const result = await inferStackConfig(`https://${userEnv.QUILT_CATALOG.replace(/^https?:\/\//, '')}`);
-            inferredEnv = result.inferredVars;
-            console.log("✓ Successfully inferred stack configuration\n");
-        } catch (error) {
-            console.error(`Warning: Could not infer configuration from catalog: ${(error as Error).message}`);
-            console.error("Falling back to environment variables only.\n");
-        }
-    }
-
-    // Merge: user env takes precedence over inferred values
-    const config = { ...inferredEnv, ...userEnv };
-
-    // Validate required user-provided values
-    const requiredUserVars = [
-        "QUILT_CATALOG",
-        "QUILT_USER_BUCKET",
-        "BENCHLING_CLIENT_ID",
-        "BENCHLING_CLIENT_SECRET",
-        "BENCHLING_TENANT",
-    ];
-
-    const missingVars = requiredUserVars.filter((varName) => !config[varName]);
-
-    if (missingVars.length > 0) {
-        console.error("Error: Missing required environment variables:");
-        missingVars.forEach((varName) => {
-            console.error(`  - ${varName}`);
-        });
-        console.error("\nPlease set these variables in your .env file.");
-        console.error("See env.template for guidance.");
-        process.exit(1);
-    }
-
-    // Validate inferred values are present (should be available if catalog lookup succeeded)
-    const requiredInferredVars = [
-        "CDK_DEFAULT_ACCOUNT",
-        "CDK_DEFAULT_REGION",
-        "QUEUE_NAME",
-        "SQS_QUEUE_URL",
-        "QUILT_DATABASE",
-    ];
-
-    const missingInferredVars = requiredInferredVars.filter((varName) => !config[varName]);
-
-    if (missingInferredVars.length > 0) {
-        console.error("Error: Could not infer required configuration:");
-        missingInferredVars.forEach((varName) => {
-            console.error(`  - ${varName}`);
-        });
-        console.error("\nThese values should be automatically inferred from your Quilt catalog.");
-        console.error("Please ensure:");
-        console.error("  1. QUILT_CATALOG is set correctly");
-        console.error("  2. Your AWS credentials have CloudFormation read permissions");
-        console.error("  3. The Quilt stack is deployed and accessible");
-        console.error("\nAlternatively, you can manually set these values in your .env file.");
-        process.exit(1);
-    }
-
-    // Validate conditional requirements
-    if (config.ENABLE_WEBHOOK_VERIFICATION !== "false" && !config.BENCHLING_APP_DEFINITION_ID) {
-        console.error("Error: BENCHLING_APP_DEFINITION_ID is required when webhook verification is enabled.");
-        console.error("Either set BENCHLING_APP_DEFINITION_ID or set ENABLE_WEBHOOK_VERIFICATION=false");
-        process.exit(1);
-    }
-
-    return config;
-}
-
-/**
- * Main execution
- */
-async function main() {
-    const config = await getConfig();
-
-    // Validate CDK bootstrap before proceeding
-    await checkCdkBootstrap(config.CDK_DEFAULT_ACCOUNT!, config.CDK_DEFAULT_REGION!);
-
-    const app = new cdk.App();
-    new BenchlingWebhookStack(app, "BenchlingWebhookStack", {
-        env: {
-            account: config.CDK_DEFAULT_ACCOUNT,
-            region: config.CDK_DEFAULT_REGION,
-        },
-        bucketName: config.QUILT_USER_BUCKET!, // User's data bucket
-        queueName: config.QUEUE_NAME!,
-        environment: "production",
-        prefix: config.PKG_PREFIX || "benchling",
-        benchlingClientId: config.BENCHLING_CLIENT_ID!,
-        benchlingClientSecret: config.BENCHLING_CLIENT_SECRET!,
-        benchlingTenant: config.BENCHLING_TENANT!,
-        quiltCatalog: config.QUILT_CATALOG!,
-        quiltDatabase: config.QUILT_DATABASE!,
-        webhookAllowList: config.WEBHOOK_ALLOW_LIST,
-        logLevel: config.LOG_LEVEL || "INFO",
-        // ECR repository configuration
-        createEcrRepository: config.CREATE_ECR_REPOSITORY === "true",
-        ecrRepositoryName: config.ECR_REPOSITORY_NAME || "quiltdata/benchling",
-    });
-}
-
-main().catch((error) => {
-    console.error("Fatal error during CDK synthesis:", error);
-    process.exit(1);
-});

package/bin/cli-auth.sh

@@ -1,74 +0,0 @@
-#!/bin/bash
-
-# Ensure environment variables are set
-if [[ -z "$BENCHLING_CLIENT_ID" || -z "$BENCHLING_TENANT" || -z "$BENCHLING_CLIENT_SECRET" ]]; then
-    echo "Error: Required environment variables are not set. Please source .env first."
-    exit 1
-fi
-
-# Debugging: Print the extracted values
-echo "BENCHLING_CLIENT_ID: $BENCHLING_CLIENT_ID"
-echo "BENCHLING_TENANT: $BENCHLING_TENANT"
-
-API_ROOT="https://${BENCHLING_TENANT}.benchling.com/api/v2"
-
-# Function to get OAuth Token
-get_token() {
-    curl -s -X POST "$API_ROOT/token" \
-        -H "Content-Type: application/x-www-form-urlencoded" \
-        -d "client_id=${BENCHLING_CLIENT_ID}" \
-        -d "client_secret=${BENCHLING_CLIENT_SECRET}" \
-        -d "grant_type=client_credentials" | jq -r '.access_token'
-}
-
-# Generic function to make API requests
-api_request() {
-    local method=$1
-    local endpoint=$2
-    local data=$3
-
-    curl -v -X "$method" "$API_ROOT/$endpoint" \
-        -H "Authorization: Bearer $TOKEN" \
-        -H "Content-Type: application/json" \
-        ${data:+--data "$data"}
-}
-
-# Get OAuth Token
-TOKEN=$(get_token)
-
-# Export TOKEN globally
-export TOKEN
-
-# Debugging: Print the token
-if [[ -z "$TOKEN" || "$TOKEN" == "null" ]]; then
-    echo "Error: Failed to retrieve access token."
-    exit 1
-fi
-echo "TOKEN: $TOKEN"
-
-# Check if CANVAS_ID is provided as an argument
-if [[ -n "$1" ]]; then
-    CANVAS_ID="$1"
-    echo "Fetching canvas with ID: $CANVAS_ID"
-    echo "=== $CANVAS_ID ==="
-    api_request "GET" "app-canvases/${CANVAS_ID}"
-    echo "=== $CANVAS_ID ==="
-
-    echo "Updating canvas with ID: $CANVAS_ID"
-    api_request "PATCH" "app-canvases/${CANVAS_ID}" '{
-        "blocks": [
-            {
-                "enabled": true,
-                "id": "user_defined_id",
-                "text": "Click me to submit",
-                "type": "BUTTON"
-            }
-        ],
-        "enabled": true,
-        "featureId": "quilt_integration"
-    }'
-
-else
-    echo "No canvas ID provided. Fetching apps instead."
-    api_request "GET" "apps"
-fi