@quiltdata/benchling-webhook 0.4.13

package/doc/PARAMETERS.md

@@ -0,0 +1,203 @@
+# CloudFormation Parameters
+
+This stack supports runtime-configurable parameters that can be updated without redeploying the entire stack.
+
+## Available Parameters
+
+### Security and Configuration
+
+#### WebhookAllowList
+
+- **Type**: String (comma-separated IP addresses)
+- **Description**: List of IP addresses allowed to send webhooks
+- **Default**: Value from `WEBHOOK_ALLOW_LIST` environment variable (or empty to allow all IPs)
+- **Example**: `34.216.192.90,34.217.183.162`
+
+#### QuiltCatalog
+
+- **Type**: String
+- **Description**: Quilt catalog URL for package links
+- **Default**: `open.quiltdata.com`
+- **Example**: `catalog.example.com`
+
+### Infrastructure Parameters
+
+#### BucketName
+
+- **Type**: String
+- **Description**: S3 bucket name for storing packages (your data bucket)
+- **Default**: Value from `QUILT_USER_BUCKET` environment variable
+- **Example**: `my-packages-bucket`
+
+#### PackagePrefix
+
+- **Type**: String
+- **Description**: Prefix for package names (no slashes allowed)
+- **Default**: Value from `PREFIX` environment variable
+- **Example**: `benchling` or `my-org`
+
+#### QueueName
+
+- **Type**: String
+- **Description**: SQS queue name for package notifications
+- **Default**: Value from `QUEUE_NAME` environment variable
+- **Example**: `my-packager-queue`
+
+**Note**: Benchling credentials (CLIENT_ID, CLIENT_SECRET, TENANT) are **not** parameters because they're used to create AWS resources at deployment time and contain sensitive information.
+
+## Updating Parameters
+
+### Using AWS CLI
+
+Update parameters without redeploying the stack:
+
+```bash
+# Update webhook allow list only
+aws cloudformation update-stack \
+  --stack-name BenchlingWebhookStack \
+  --use-previous-template \
+  --parameters \
+    ParameterKey=WebhookAllowList,ParameterValue="1.2.3.4,5.6.7.8" \
+    ParameterKey=QuiltCatalog,UsePreviousValue=true \
+    ParameterKey=BucketName,UsePreviousValue=true \
+    ParameterKey=PackagePrefix,UsePreviousValue=true \
+    ParameterKey=QueueName,UsePreviousValue=true
+
+# Update multiple parameters
+aws cloudformation update-stack \
+  --stack-name BenchlingWebhookStack \
+  --use-previous-template \
+  --parameters \
+    ParameterKey=WebhookAllowList,ParameterValue="34.216.192.90,34.217.183.162" \
+    ParameterKey=QuiltCatalog,ParameterValue="my-catalog.quiltdata.com" \
+    ParameterKey=BucketName,ParameterValue="new-bucket-name" \
+    ParameterKey=PackagePrefix,UsePreviousValue=true \
+    ParameterKey=QueueName,UsePreviousValue=true
+```
+
+### Using AWS Console
+
+1. Go to CloudFormation in AWS Console
+2. Select the `BenchlingWebhookStack` stack
+3. Click **Update**
+4. Select **Use current template**
+5. Click **Next**
+6. Update the parameter values as needed:
+   - **WebhookAllowList**: Comma-separated IP addresses
+   - **QuiltCatalog**: Catalog URL
+   - **BucketName**: S3 bucket name
+   - **PackagePrefix**: Package prefix
+   - **QueueName**: SQS queue name
+7. Click **Next** through the remaining screens
+8. Click **Update stack**
+
+### Using CDK
+
+Parameters are automatically created during CDK deployment. The initial values come from environment variables, but subsequent updates can be done via CloudFormation (as shown above) without redeploying via CDK.
+
+## Benefits
+
+- **No Code Deployment**: Update security settings (IP allowlist) without redeploying code
+- **Fast Updates**: CloudFormation parameter updates are much faster than full stack updates
+- **Zero Downtime**: Parameter updates don't require Lambda redeployment
+- **Audit Trail**: All parameter changes are tracked in CloudFormation change sets
+
+## Implementation Details
+
+### Architecture
+
+1. **CloudFormation Parameters**: Defined at stack level
+2. **Lambda Environment Variables**: Automatically updated from parameters
+3. **API Gateway Resource Policy**: Automatically updated from parameters
+
+### How It Works
+
+The stack uses a hybrid approach:
+
+- **Initial Deployment**: Uses environment variable values from `bin/benchling-webhook.ts`
+- **Subsequent Updates**: Uses CloudFormation parameter values
+- **Lambda Environment**: Automatically reflects parameter changes via CloudFormation
+
+When you update a parameter via CloudFormation:
+
+1. CloudFormation updates the Lambda function's environment variables
+2. CloudFormation updates the API Gateway Resource Policy
+3. Lambda automatically picks up the new values on next invocation (no cold start required)
+4. API Gateway immediately enforces the new IP allowlist
+
+## Examples
+
+### Add a new IP to the allowlist
+
+```bash
+# Current IPs: 34.216.192.90,34.217.183.162
+# Adding new IP: 203.0.113.10
+
+aws cloudformation update-stack \
+  --stack-name BenchlingWebhookStack \
+  --use-previous-template \
+  --parameters \
+    ParameterKey=WebhookAllowList,ParameterValue="34.216.192.90,34.217.183.162,203.0.113.10" \
+    ParameterKey=QuiltCatalog,UsePreviousValue=true \
+    ParameterKey=BucketName,UsePreviousValue=true \
+    ParameterKey=PackagePrefix,UsePreviousValue=true \
+    ParameterKey=QueueName,UsePreviousValue=true
+```
+
+### Switch to a different S3 bucket
+
+```bash
+aws cloudformation update-stack \
+  --stack-name BenchlingWebhookStack \
+  --use-previous-template \
+  --parameters \
+    ParameterKey=WebhookAllowList,UsePreviousValue=true \
+    ParameterKey=QuiltCatalog,UsePreviousValue=true \
+    ParameterKey=BucketName,ParameterValue="new-packages-bucket" \
+    ParameterKey=PackagePrefix,UsePreviousValue=true \
+    ParameterKey=QueueName,UsePreviousValue=true
+```
+
+### Change package prefix
+
+```bash
+aws cloudformation update-stack \
+  --stack-name BenchlingWebhookStack \
+  --use-previous-template \
+  --parameters \
+    ParameterKey=WebhookAllowList,UsePreviousValue=true \
+    ParameterKey=QuiltCatalog,UsePreviousValue=true \
+    ParameterKey=BucketName,UsePreviousValue=true \
+    ParameterKey=PackagePrefix,ParameterValue="my-new-prefix" \
+    ParameterKey=QueueName,UsePreviousValue=true
+```
+
+### Allow all IPs (remove IP filtering)
+
+```bash
+aws cloudformation update-stack \
+  --stack-name BenchlingWebhookStack \
+  --use-previous-template \
+  --parameters \
+    ParameterKey=WebhookAllowList,ParameterValue="" \
+    ParameterKey=QuiltCatalog,UsePreviousValue=true \
+    ParameterKey=BucketName,UsePreviousValue=true \
+    ParameterKey=PackagePrefix,UsePreviousValue=true \
+    ParameterKey=QueueName,UsePreviousValue=true
+```
+
+## Troubleshooting
+
+### Parameter update fails
+
+If the CloudFormation update fails, check:
+
+- IP addresses are valid (no spaces, proper CIDR notation if needed)
+- Catalog URL is a valid hostname
+- You're using `--use-previous-template` flag
+
+### Changes not taking effect
+
+- API Gateway Resource Policy updates are immediate
+- Lambda environment variable updates require the Lambda to be invoked again
+- Check CloudFormation Events tab for update status

package/doc/RELEASE.md

@@ -0,0 +1,297 @@
+# Release Process
+
+This document describes how to create releases for the benchling-webhook project.
+
+## Overview
+
+The project uses an automated CI/CD pipeline that:
+
+1. Runs tests on all pushes and pull requests
+2. Builds and pushes Docker images when tags are created
+3. Creates GitHub releases with auto-generated release notes
+4. Publishes to NPM and GitHub Packages
+
+## Quick Start
+
+### Create a Development Release (for testing)
+
+```bash
+npm run version:dev
+# This creates version 0.4.8-dev.0 (or bumps existing dev version)
+# Then push the tag:
+git push origin v0.4.8-dev.0
+```
+
+### Create a Production Release
+
+```bash
+# For patch releases (0.4.7 -> 0.4.8)
+npm run version:patch
+
+# For minor releases (0.4.7 -> 0.5.0)
+npm run version:minor
+
+# For major releases (0.4.7 -> 1.0.0)
+npm run version:major
+
+# Then push the tag:
+git push origin v0.4.8
+```
+
+## Detailed Workflow
+
+### 1. Prepare Your Release
+
+Before creating a release:
+
+1. **Ensure all changes are committed and pushed**
+
+   ```bash
+   git status  # Should show "nothing to commit, working tree clean"
+   ```
+
+2. **Update CHANGELOG.md** (optional but recommended)
+
+   ```markdown
+   ## [0.4.8] - 2025-10-27
+
+   ### Added
+   - New feature description
+
+   ### Changed
+   - Updated component behavior
+
+   ### Fixed
+   - Bug fix description
+   ```
+
+3. **Ensure you're on the main branch**
+
+   ```bash
+   git checkout main
+   git pull origin main
+   ```
+
+### 2. Create a Release
+
+#### Development Releases (for testing)
+
+Development releases are marked as "pre-release" in GitHub and are NOT published to NPM.
+
+```bash
+# Create a new dev release
+npm run version:dev
+
+# Or bump an existing dev release
+npm run version:dev-bump
+```
+
+This will:
+
+- Update `package.json` with the new version
+- Commit the version change
+- Create a git tag (e.g., `v0.4.8-dev.0`)
+- Show instructions for pushing the tag
+
+#### Production Releases
+
+```bash
+# Choose the appropriate bump type:
+npm run version:patch  # For bug fixes (0.4.7 -> 0.4.8)
+npm run version:minor  # For new features (0.4.7 -> 0.5.0)
+npm run version:major  # For breaking changes (0.4.7 -> 1.0.0)
+```
+
+### 3. Push the Tag
+
+After creating the tag, push it to trigger the CI/CD pipeline:
+
+```bash
+git push origin v0.4.8  # Replace with your actual tag
+```
+
+### 4. Monitor the Release
+
+1. **Watch the GitHub Actions workflow**: <https://github.com/quiltdata/benchling-webhook/actions>
+2. **Check the workflow progress**:
+   - Test job runs first (Python + Node.js tests)
+   - Docker job runs after tests pass (builds and pushes image)
+   - Release job creates GitHub release and publishes packages
+
+3. **Verify the release**:
+   - GitHub Release: <https://github.com/quiltdata/benchling-webhook/releases>
+   - Docker Image in ECR: `{account-id}.dkr.ecr.us-east-1.amazonaws.com/quiltdata/benchling:{version}`
+   - NPM Package: <https://www.npmjs.com/package/quilt-benchling-webhook>
+   - GitHub Package: <https://github.com/quiltdata/benchling-webhook/pkgs/npm/quilt-benchling-webhook>
+
+## CI/CD Pipeline Details
+
+### Workflow Triggers
+
+The CI workflow (`.github/workflows/ci.yaml`) is triggered by:
+
+- **Push to main**: Runs tests only
+- **Pull requests**: Runs tests only
+- **Tags matching `v*.*.*`**: Runs tests, builds Docker, creates release
+- **Tags matching `v*.*.*-dev.*`**: Same as above, but marks as pre-release
+
+### Jobs
+
+1. **test** (always runs)
+   - Sets up Node.js 22 and Python 3.12
+   - Installs dependencies
+   - Runs Python tests (pytest)
+   - Runs Node.js tests (jest)
+   - Builds the package
+
+2. **docker** (only on tags, after test passes)
+   - Builds Docker image using `make push-ci`
+   - Pushes to ECR with version tag and `latest` tag
+   - Outputs image URI for release notes
+
+3. **release** (only on tags, after docker completes)
+   - Generates release notes from CHANGELOG.md
+   - Creates GitHub release (pre-release for dev tags)
+   - Publishes to NPM (production releases only)
+   - Publishes to GitHub Packages
+
+## Version Management Script
+
+The `bin/version.js` script handles version bumping and tagging:
+
+```bash
+# View current version and help
+npm run release
+
+# Bump versions
+npm run version:major      # 0.4.7 -> 1.0.0
+npm run version:minor      # 0.4.7 -> 0.5.0
+npm run version:patch      # 0.4.7 -> 0.4.8
+npm run version:dev        # 0.4.7 -> 0.4.8-dev.0 or 0.4.8-dev.0 -> 0.4.8-dev.1
+npm run version:dev-bump   # 0.4.8-dev.0 -> 0.4.8-dev.1
+
+# Update version without creating tag
+node bin/version.js patch --no-tag
+```
+
+## Docker Images
+
+Docker images are automatically built and pushed to ECR with:
+
+- Version tag: `quiltdata/benchling:0.4.8`
+- Latest tag: `quiltdata/benchling:latest`
+
+To pull and run:
+
+```bash
+docker pull {account-id}.dkr.ecr.us-east-1.amazonaws.com/quiltdata/benchling:0.4.8
+docker run -p 5000:5000 --env-file .env {account-id}.dkr.ecr.us-east-1.amazonaws.com/quiltdata/benchling:0.4.8
+```
+
+## Troubleshooting
+
+### Tag already exists
+
+```bash
+# Delete local tag
+git tag -d v0.4.8
+
+# Delete remote tag (if pushed)
+git push origin :refs/tags/v0.4.8
+```
+
+### Workflow failed
+
+1. Check the GitHub Actions logs
+2. Common issues:
+   - Tests failing: Fix tests and create a new tag
+   - Docker build failing: Check `docker/` directory and Makefile
+   - AWS credentials: Verify secrets are configured
+   - ECR repository doesn't exist: Run `make -C docker docker-ecr-create`
+
+### Need to re-release
+
+1. Delete the tag (locally and remotely)
+2. Delete the GitHub release
+3. Create a new tag with the same or different version
+
+## Manual NPM Publishing
+
+In cases where you need to manually publish to NPM (e.g., CI/CD is unavailable, emergency hotfix, or testing), you can use the manual publish script:
+
+### Prerequisites
+
+1. **NPM Access Token**: You need an NPM access token with publish permissions
+   - Visit: <https://www.npmjs.com/settings/[your-username]/tokens>
+   - Click "Generate New Token"
+   - Choose "Automation" (for CI/CD) or "Publish" (for manual use)
+   - Copy the token (it starts with `npm_`)
+
+### Usage
+
+```bash
+# Test the publish process (dry-run)
+NPM_TOKEN=npm_xxxxx npm run publish:manual -- --dry-run
+
+# Publish to NPM
+NPM_TOKEN=npm_xxxxx npm run publish:manual
+
+# Publish with a specific dist-tag
+NPM_TOKEN=npm_xxxxx npm run publish:manual -- --tag beta
+
+# View help
+npm run publish:manual -- --help
+```
+
+### How It Works
+
+The script:
+1. Validates your NPM token
+2. Checks for uncommitted changes (with confirmation prompt)
+3. Creates a temporary `.npmrc` file with your token
+4. Runs `npm publish --access public`
+5. Cleans up the temporary `.npmrc` file
+6. Restores any existing `.npmrc` backup
+
+### Security Notes
+
+- The script creates `.npmrc` with restricted permissions (0600)
+- Your token is never committed to git (`.npmrc` is in `.gitignore`)
+- The temporary `.npmrc` is automatically cleaned up, even on errors
+- Always use `--dry-run` first to test before publishing
+
+## NPM Scripts Reference
+
+| Script | Description |
+|--------|-------------|
+| `npm run release` | Show current version and help |
+| `npm run version:major` | Bump major version (breaking changes) |
+| `npm run version:minor` | Bump minor version (new features) |
+| `npm run version:patch` | Bump patch version (bug fixes) |
+| `npm run version:dev` | Create or bump dev version |
+| `npm run version:dev-bump` | Bump dev counter only |
+| `npm run docker-push` | Build and push Docker image locally |
+| `npm run docker-check` | Validate Docker images in registry |
+| `npm run publish:manual` | Manually publish to NPM using access token |
+
+## Best Practices
+
+1. **Always test before releasing**: Run `npm test` and manual testing
+2. **Use dev releases for testing**: Test the CI/CD pipeline with dev tags first
+3. **Update CHANGELOG.md**: Document changes before releasing
+4. **Follow semantic versioning**:
+   - Major: Breaking changes
+   - Minor: New features (backward compatible)
+   - Patch: Bug fixes (backward compatible)
+5. **Review the release**: Check GitHub release notes and test the published packages
+6. **Coordinate with team**: Communicate releases in team channels
+
+## Migration Notes
+
+The previous release process using `bin/create-release.sh` and separate `release.yaml` workflow has been replaced with this integrated CI/CD pipeline. Key changes:
+
+- **Single workflow**: All release steps are now in `.github/workflows/ci.yaml`
+- **Automated**: No manual Docker builds or release note creation needed
+- **Tag-based**: Simply push a tag to trigger the release
+- **Pre-release support**: Dev tags are automatically marked as pre-releases
+- **Integrated testing**: Docker images are only built after tests pass

package/doc/RELEASE_NOTES.md

@@ -0,0 +1,64 @@
+# Benchling Webhook Integration for Quilt
+
+Connects Benchling lab notebook entries to Quilt data packages via webhooks.
+
+## Quick Install
+
+**Prerequisites:** AWS account, Node.js 18+, Docker, existing Quilt deployment
+
+```bash
+# 1. Clone and install
+git clone https://github.com/quiltdata/benchling-webhook.git
+cd benchling-webhook
+npm install
+
+# 2. Configure (auto-infer from Quilt catalog)
+npm run get-env -- https://quilt-catalog.yourcompany.com --write
+cp env.inferred .env
+# Edit .env to add Benchling credentials
+
+# 3. Deploy
+source .env
+npx cdk bootstrap aws://$CDK_DEFAULT_ACCOUNT/$CDK_DEFAULT_REGION  # first time only
+npm run deploy
+
+# 4. Configure Benchling app
+# - Create app from app-manifest.yaml
+# - Set webhook URL from .env.deploy
+# - Install and activate
+
+# 5. Verify
+source .env.deploy
+curl $WEBHOOK_ENDPOINT/health
+```
+
+## Usage
+
+1. Create entry in Benchling
+2. Insert Canvas → "Quilt Integration"
+3. Click "Create" to make package
+4. Add files and click "Update package"
+
+## Documentation
+
+- [AGENTS.md](AGENTS.md) - Complete deployment guide, architecture, configuration
+- [docker/README.md](docker/README.md) - Development workflows
+- [doc/RELEASE.md](doc/RELEASE.md) - Release process
+
+## License
+
+Apache-2.0
+
+---
+
+## Docker Images (for custom deployments)
+
+**Latest Release (v0.4.12):**
+- Production: `public.ecr.aws/quiltdata/benchling:0.4.12`
+- Latest: `public.ecr.aws/quiltdata/benchling:latest`
+
+Pull and run:
+```bash
+docker pull public.ecr.aws/quiltdata/benchling:0.4.12
+docker run -p 5000:5000 --env-file .env public.ecr.aws/quiltdata/benchling:0.4.12
+```

package/env.template

@@ -0,0 +1,67 @@
+# ==============================================================================
+# REQUIRED USER VALUES
+# ==============================================================================
+# These are the ONLY values you need to provide. Everything else is inferred
+# from your Quilt catalog configuration at deployment time.
+
+# ==============================================================================
+# Quilt Configuration
+# ==============================================================================
+
+# Your Quilt catalog URL (just the domain, without https://)
+QUILT_CATALOG=quilt-catalog.yourcompany.com
+
+# The S3 bucket where you want to store Benchling exports
+# This is YOUR data bucket, not the Quilt system buckets
+QUILT_USER_BUCKET=your-data-bucket
+
+# ==============================================================================
+# Benchling Configuration
+# ==============================================================================
+# BENCHLING_TENANT: Use XXX if you login to Benchling at XXX.benchling.com
+
+BENCHLING_TENANT=your-tenant
+BENCHLING_APP_DEFINITION_ID=appdef_your_id_here
+BENCHLING_CLIENT_ID=your-client-id
+BENCHLING_CLIENT_SECRET=your-client-secret
+
+# BENCHLING_API_KEY=your-api-key  # not currently used
+
+# ==============================================================================
+# OPTIONAL VALUES (Override defaults if needed)
+# ==============================================================================
+
+
+# Package Configuration
+PKG_PREFIX=benchling   # Prefix for Quilt package names (e.g., "benchling/my-experiment")
+PKG_KEY=experiment_id  # Metadata key used to link Benchling entries to packages
+
+
+# Webhook Security
+ENABLE_WEBHOOK_VERIFICATION=true  # Set to false ONLY for local development
+
+# Logging (optional - defaults to INFO)
+# LOG_LEVEL=DEBUG  # Options: DEBUG, INFO, WARNING, ERROR, CRITICAL
+
+# Test Configuration (for running tests)
+BENCHLING_TEST_ENTRY=etr_123456789
+
+# Ngrok Configuration (optional - for local development)
+# Get your authtoken from: https://dashboard.ngrok.com/get-started/your-authtoken
+# NGROK_AUTHTOKEN=your_ngrok_authtoken_here
+# NGROK_DOMAIN=your-subdomain.ngrok-free.app
+
+# ==============================================================================
+# AUTOMATICALLY INFERRED VALUES
+# ==============================================================================
+# The following values are automatically inferred from your Quilt catalog
+# at deployment time via bin/get-env.js:
+#
+# - CDK_DEFAULT_ACCOUNT    (from AWS STS)
+# - CDK_DEFAULT_REGION     (from catalog config)
+# - QUEUE_NAME             (from Quilt stack outputs)
+# - SQS_QUEUE_URL          (from Quilt stack outputs)
+# - QUILT_DATABASE         (from Quilt stack outputs)
+#
+# You do NOT need to set these unless you want to override the inferred values.
+# ==============================================================================

package/jest.config.js

@@ -0,0 +1,14 @@
+module.exports = {
+  testEnvironment: 'node',
+  roots: ['<rootDir>/test'],
+  testMatch: ['**/*.test.ts'],
+  transform: {
+    '^.+\\.tsx?$': ['ts-jest', {
+      tsconfig: {
+        module: 'node20',
+        resolveJsonModule: true
+      }
+    }]
+  },
+  cacheDirectory: '<rootDir>/.jest-cache'
+};

package/lib/README.md

@@ -0,0 +1,50 @@
+# Benchling Webhook Stack Architecture
+
+This directory contains the core CDK infrastructure code split across multiple files for better organization:
+
+## Core Components
+
+### benchling-webhook-stack.ts
+
+The main stack file that orchestrates the entire infrastructure. It:
+
+- Creates the S3 bucket reference
+- Sets up the Benchling OAuth connection
+- Instantiates the state machine and API gateway
+- Manages stack outputs
+
+### state-machine.ts
+
+Contains the Step Functions state machine definition that:
+
+- Processes incoming webhook events
+- Stores event data in S3
+- Fetches entry details from Benchling API
+- Sends notifications to SQS
+
+### api-gateway.ts
+
+Handles the API Gateway configuration including:
+
+- REST API setup with logging
+- IAM roles and permissions
+- Webhook endpoints for different event types
+- Integration with Step Functions
+
+## Flow
+
+1. API Gateway receives webhook POST requests
+2. Requests trigger Step Functions execution
+3. State machine:
+   - Stores raw event in S3
+   - Fetches entry details from Benchling
+   - Stores entry data in S3
+   - Sends notification to SQS
+
+## Key Features
+
+- Modular architecture for maintainability
+- Separation of concerns between components
+- Reusable constructs
+- Proper error handling
+- Comprehensive logging