superset-showtime 0.2.8__tar.gz → 0.3.1__tar.gz

{superset_showtime-0.2.8 → superset_showtime-0.3.1}/.claude/settings.local.json

@@ -30,7 +30,11 @@
       "Bash(git grep:*)",
       "Bash(git push:*)",
       "Bash(make lint:*)",
-      "Bash(mypy:*)"
+      "Bash(mypy:*)",
+      "Bash(sed:*)",
+      "Bash(git pull:*)",
+      "Bash(git rebase:*)",
+      "Bash(git fetch:*)"
     ],
     "deny": [],
     "ask": []

{superset_showtime-0.2.8 → superset_showtime-0.3.1}/CLAUDE.md

@@ -4,7 +4,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 
 ## Project Overview
 
-Superset Showtime is a Python CLI tool for managing Apache Superset ephemeral environments using **circus tent emoji labels** as a state management system on GitHub PRs. The system replaces complex GitHub Actions scripts with a simple CLI that uses GitHub labels for ACID-like state transactions.
+Superset Showtime is a Python CLI tool for managing Apache Superset ephemeral environments using **circus tent emoji labels** as a state management system on GitHub PRs. The system implements **ACID-style atomic transactions** using GitHub labels as a distributed coordination mechanism, with **compare-and-swap** patterns to prevent race conditions.
 
 ## Development Commands
 
@@ -22,11 +22,11 @@ python -m showtime labels               # Complete label reference
 python -m showtime list                 # List all environments (requires GITHUB_TOKEN)
 python -m showtime test-lifecycle 1234  # Full workflow simulation
 
-# Test with real GitHub PRs safely
+# Test new unified sync command
 export GITHUB_TOKEN=xxx
-showtime start 34809 --dry-run-aws --aws-sleep 5    # Mock AWS, real GitHub labels
-showtime status 34809                                # Check environment status
-showtime stop 34809 --force                         # Clean up test labels
+showtime sync 34809 --check-only                    # Analyze what's needed
+showtime sync 34809 --dry-run-aws --dry-run-docker # Test full flow safely
+showtime set-state building 34809                   # Manual state transitions
 ```
 
 ### Testing Single Components
@@ -39,10 +39,19 @@ python -c "from showtime.core.circus import Show; show = Show(pr_number=1234, sh
 ## Architecture Overview
 
 ### Core State Management Pattern
-The system uses **GitHub labels as a distributed state machine**:
-- **Trigger labels** (`🎪 trigger-start`) - Commands added by users, processed and removed by CLI
-- **State labels** (`🎪 🚦 abc123f running`) - Per-SHA status managed by CLI
+The system uses **GitHub labels as a distributed ACID-style database**:
+- **Trigger labels** (`🎪 ⚡ showtime-trigger-start`) - Commands added by users, atomically processed and removed
+- **State labels** (`🎪 abc123f 🚦 building`) - Per-SHA status managed with atomic compare-and-swap operations
 - **No external database** - All state reconstructed from GitHub labels
+- **Race condition prevention** - Atomic claim prevents double-processing of triggers
+
+### Enhanced State Lifecycle
+**Complete state progression**: `building → built → deploying → running/failed`
+- **building**: Docker image construction in progress
+- **built**: Docker build complete, ready for AWS deployment
+- **deploying**: AWS ECS deployment in progress
+- **running**: Environment live and accessible
+- **failed**: Build or deployment failed
 
 ### Key Classes and Responsibilities
 
@@ -96,8 +105,9 @@ Replicates current GitHub Actions AWS logic:
 - **Security model**: `pull_request_target` + PyPI install (no PR code execution)
 
 #### CLI Commands Called by GHA
-- `showtime handle-trigger {pr-number}` - Process trigger labels
-- `showtime handle-sync {pr-number}` - Handle new commits
+- `showtime sync {pr-number} --check-only` - Analyze state and determine build_needed
+- `showtime sync {pr-number} --sha {sha}` - Execute atomic claim + build + deploy
+- `showtime set-state {state} {pr-number}` - Manual state transitions
 - `showtime cleanup --older-than 48h` - Scheduled cleanup
 
 ## Important Implementation Details
@@ -114,15 +124,23 @@ The CLI has comprehensive dry-run support for safe testing:
 ```bash
 --dry-run-aws        # Skip AWS operations, use mock data
 --dry-run-github     # Skip GitHub operations, show what would happen
+--dry-run-docker     # Skip Docker build, use mock success
 --aws-sleep N        # Sleep N seconds to simulate AWS timing
 ```
 
-### AWS Resource Naming
-Must maintain compatibility with existing Superset infrastructure:
+### AWS Resource Naming & Docker Integration
+**Direct Docker build** (no supersetbot dependency):
+- **Docker Image**: `apache/superset:pr-{pr_number}-{sha}-ci` (single tag, built directly)
 - **ECS Service**: `pr-{pr_number}-{sha}` (e.g., `pr-1234-abc123f`)
-- **ECR Image**: `pr-{pr_number}-{sha}-ci`
 - **Network**: Same subnets/security groups as current production setup
 
+**Docker Build Command**:
+```bash
+docker buildx build --push --load --platform linux/amd64 --target ci \
+  --build-arg INCLUDE_CHROMIUM=false --build-arg LOAD_EXAMPLES_DUCKDB=true \
+  -t apache/superset:pr-{pr}-{sha}-ci .
+```
+
 ### State Recovery Pattern
 Since the CLI is stateless, it always reconstructs state from GitHub labels:
 ```python
@@ -133,15 +151,22 @@ show = pr.current_show                       # Finds active environment
 
 ## Critical Development Notes
 
-### Label Operations Must Be Atomic
-GitHub label operations are the "database transactions" - handle carefully:
+### ACID-Style Atomic Transactions
+GitHub label operations implement compare-and-swap pattern for race condition prevention:
 ```python
-# Remove trigger immediately after processing
-github.remove_label(pr_number, trigger_label)
-# Update state labels atomically
-github.remove_circus_labels(pr_number)
-for label in new_labels:
-    github.add_label(pr_number, label)
+# Atomic claim pattern in _atomic_claim_environment():
+# 1. CHECK: Validate current state allows new work
+# 2. COMPARE: Ensure triggers exist and state is valid
+# 3. SWAP: Remove triggers + Set building state atomically
+# 4. COMMIT: Environment successfully claimed
+
+# Example atomic transaction:
+if not _validate_non_active_state(pr):
+    return False  # Another job already active
+github.remove_label(pr_number, trigger_label)  # Remove trigger
+github.remove_circus_labels(pr_number)         # Clear stale state
+for label in building_show.to_circus_labels():
+    github.add_label(pr_number, label)         # Set building state
 ```
 
 ### Per-SHA Format Required
@@ -166,16 +191,14 @@ showtime stop 34809 --force
 
 ## Current Implementation Status
 
-### ✅ Fully Implemented (Production Ready)
-- **Blue-green deployment**: Zero-downtime updates with health checks
-- **AWS integration**: Complete ECS/ECR operations with DockerHub pulling
-- **Smart sync system**: Intelligent PR state detection and auto-sync
-- **GitHub Actions workflows**: Drop-in replacement for ephemeral-env.yml
-- **TTL-based cleanup**: Respects individual environment preferences
-- **SHA override support**: Deploy any specific commit for testing
-- **Freeze functionality**: Pin environments during testing
-- **Enhanced CLI**: Clickable links, full-width tables, real-time progress
-- **Unified label system**: Searchable namespaced labels with color themes
+### ✅ Current Architecture (Production Ready)
+- **ACID-style transactions**: Atomic compare-and-swap prevents race conditions
+- **Direct Docker integration**: No supersetbot dependency, single tag builds
+- **Streamlined GitHub Actions**: 3-step workflow (check → setup → sync)
+- **Enhanced state machine**: building→built→deploying→running/failed lifecycle
+- **Unified sync command**: Handles atomic claim + build + deploy in one command
+- **Smart conditionals**: Skip Docker setup when build_needed=false
+- **Race condition safe**: Multiple jobs can't double-process triggers
 
 ### 🎯 Label System (Streamlined)
 - **Trigger labels**: `🎪 ⚡ showtime-trigger-start` (namespaced, searchable)
@@ -183,8 +206,16 @@ showtime stop 34809 --force
 - **Freeze support**: `🎪 🧊 showtime-freeze` (prevents auto-sync)
 - **Automatic creation**: Labels get proper colors/descriptions automatically
 
-### 📦 Ready for Deployment
-- **GitHub Actions**: `workflows-reference/showtime-trigger.yml` + `showtime-cleanup.yml`
-- **PyPI package**: Built with dependencies, ready for `pip install superset-showtime`
-- **Testing infrastructure**: Comprehensive dry-run and manual testing support
-- **Documentation**: Complete README with workflows and examples
+### 📦 Key Commands for Development
+```bash
+# Core sync command (replaces handle-trigger, handle-sync):
+showtime sync PR_NUMBER --check-only        # Returns build_needed + target_sha
+showtime sync PR_NUMBER --sha SHA           # Atomic claim + build + deploy
+
+# Manual state management:
+showtime set-state building PR_NUMBER       # Set specific state
+showtime set-state failed PR_NUMBER --error-msg "Build failed"
+
+# Development testing:
+showtime sync PR_NUMBER --dry-run-aws --dry-run-docker --dry-run-github
+```

{superset_showtime-0.2.8 → superset_showtime-0.3.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: superset-showtime
-Version: 0.2.8
+Version: 0.3.1
 Summary: 🎪 Apache Superset ephemeral environment management with circus tent emoji state tracking
 Project-URL: Homepage, https://github.com/apache/superset-showtime
 Project-URL: Documentation, https://superset-showtime.readthedocs.io/
@@ -113,9 +113,34 @@ Superset Showtime is a CLI tool designed primarily for **GitHub Actions** to man
 🎪 abc123f 🌐 52-1-2-3      # Available at http://52.1.2.3:8080
 ```
 
-## 📊 For Maintainers (CLI Operations)
+### 🔄 Showtime Workflow
+
+```mermaid
+flowchart TD
+    A[User adds 🎪 ⚡ trigger-start] --> B[GitHub Actions: sync]
+    B --> C{Current state?}
+
+    C -->|No environment| D[🔒 Claim: Remove trigger + Set building]
+    C -->|Running + new SHA| E[🔒 Claim: Remove trigger + Set building]
+    C -->|Already building| F[❌ Exit: Another job active]
+    C -->|No triggers| G[❌ Exit: Nothing to do]
+
+    D --> H[📋 State: building]
+    E --> H
+    H --> I[🐳 Docker build]
+    I -->|Success| J[📋 State: built]
+    I -->|Fail| K[📋 State: failed]
+
+    J --> L[📋 State: deploying]
+    L --> M[☁️ AWS Deploy]
+    M -->|Success| N[📋 State: running]
+    M -->|Fail| O[📋 State: failed]
+
+    N --> P[🎪 Environment ready!]
+
+    Q[User adds 🎪 🛑 trigger-stop] --> R[🧹 Cleanup AWS + Remove labels]
+```
 
-> **Note**: CLI is mainly for debugging or developing Showtime itself. Primary interface is GitHub labels above.
 
 **Install CLI for debugging:**
 ```bash
@@ -229,129 +254,31 @@ You'll see:
 
 ### GitHub Actions Integration
 
-Showtime is designed to be called by Superset's GitHub Actions workflows:
-
-```yaml
-# .github/workflows/showtime.yml - Integrates with Superset's existing build workflows
-on:
-  pull_request_target:
-    types: [labeled, unlabeled, synchronize]
-
-jobs:
-  showtime-handler:
-    if: contains(github.event.label.name, '🎪')
-    steps:
-      - name: Install Showtime from PyPI
-        run: pip install superset-showtime
-
-      - name: Process circus triggers
-        run: python -m showtime handle-trigger ${{ github.event.pull_request.number }}
-```
-
-**Integration approach:**
-- **Coordinates with Superset builds** - Uses existing container build workflows
-- **Runs trusted code** (from PyPI, not PR code)
-- **Simple orchestration logic** (install CLI and run commands)
-- **Leverages existing infrastructure** - Same AWS resources and permissions
-
-## 🛠️ Installation & Setup
-
-### For Contributors (GitHub Labels Only)
-No installation needed! Just use GitHub labels to trigger environments.
-
-### For Maintainers (Manual CLI Operations)
+**🎯 Live Workflow**: [showtime-trigger.yml](https://github.com/apache/superset/actions/workflows/showtime-trigger.yml)
 
-**Install CLI for debugging/testing:**
-```bash
-pip install superset-showtime
-export GITHUB_TOKEN=your_personal_access_token
-```
+**How it works:**
+- Triggers on PR label changes, commits, and closures
+- Installs `superset-showtime` from PyPI (trusted code, not PR code)
+- Runs `showtime sync` to handle trigger processing and deployments
+- Supports manual testing via `workflow_dispatch` with specific SHA override
 
-**Manual operations:**
+**Commands used:**
 ```bash
-showtime list                    # Monitor all active environments
-showtime status 1234            # Debug specific environment
-showtime labels                 # Reference complete label system
+showtime sync PR_NUMBER --check-only    # Determine build_needed + target_sha
+showtime sync PR_NUMBER --sha SHA       # Execute atomic claim + build + deploy
 ```
 
-### For Repository Integration (GitHub Actions)
-
-**1. Install GitHub workflows:**
-Copy `workflows-reference/showtime-trigger.yml` and `workflows-reference/showtime-cleanup.yml` to Superset's `.github/workflows/`.
-
-**2. Configure secrets (already exist in Superset):**
-- `AWS_ACCESS_KEY_ID`
-- `AWS_SECRET_ACCESS_KEY`
-- `GITHUB_TOKEN`
-
-**3. Dependencies:**
-Showtime coordinates with Superset's existing build infrastructure - no additional setup needed.
-
-## 📊 CLI Reference (For Development/Debugging)
+## 🛠️ CLI Usage
 
-> **Primary Interface**: Use GitHub labels in PR interface. CLI is mainly for maintainers debugging or developing Showtime itself.
+The CLI is primarily used by GitHub Actions, but available for debugging and advanced users:
 
-### Debugging Commands
 ```bash
-showtime list                         # Monitor all environments
-showtime status 1234                  # Debug specific environment
-showtime labels                       # Complete label reference
-showtime test-lifecycle 1234          # Full workflow simulation
-```
-
-### Manual Operations (Advanced)
-```bash
-showtime start 1234                    # Manually create environment
-showtime start 1234 --sha abc123f     # Create environment (specific SHA)
-showtime stop 1234                    # Manually delete environment
-showtime sync 1234                    # Force sync to desired state
-showtime cleanup --respect-ttl        # Manual cleanup
+pip install superset-showtime
+export GITHUB_TOKEN=your_token
+showtime --help  # See all available commands
 ```
 
-### GitHub Actions Commands
-```bash
-showtime handle-trigger 1234          # Process trigger labels (called by GHA)
-showtime cleanup --older-than 48h     # Scheduled cleanup (called by GHA)
-```
 
-## 🎪 Benefits for Superset
-
-### For Contributors
-- **🎯 Simple workflow** - Just add/remove GitHub labels
-- **👀 Visual feedback** - See environment status in PR labels
-- **⚡ Automatic updates** - New commits update environments automatically
-- **🔧 Configuration testing** - Test config changes through code commits
-
-### For Maintainers
-- **📊 Complete visibility** - `showtime list` shows all environments
-- **🧹 Easy cleanup** - Automatic expired environment cleanup
-- **🔍 Better debugging** - Clear state in labels, comprehensive CLI
-- **💰 Cost savings** - No duplicate environments, proper cleanup
-
-### For Operations
-- **📝 Simpler workflows** - Replace complex GHA scripts with simple CLI calls
-- **🔒 Same security model** - No new permissions needed
-- **🎯 Deterministic** - Predictable AWS resource naming
-- **🚨 Monitoring ready** - 48h maximum lifetime, scheduled cleanup
-
-## 🏗️ Architecture
-
-### State Management
-All state lives in **GitHub labels** - no external databases needed:
-- **Trigger labels** (`🎪 trigger-*`) - Commands that get processed and removed
-- **State labels** (`🎪 🚦 *`) - Current environment status, managed by CLI
-
-### AWS Resources
-Deterministic naming enables reliable cleanup:
-- **ECS Service:** `pr-{pr_number}-{sha}` (e.g., `pr-1234-abc123f`)
-- **ECR Image:** `pr-{pr_number}-{sha}-ci` (e.g., `pr-1234-abc123f-ci`)
-
-### Rolling Updates
-Zero-downtime updates by running multiple environments:
-1. Keep old environment serving traffic
-2. Build new environment in parallel
-3. Switch traffic when new environment is healthy
-4. Clean up old environment
 
 ## 🤝 Contributing
 

{superset_showtime-0.2.8 → superset_showtime-0.3.1}/README.md

@@ -48,9 +48,34 @@ Superset Showtime is a CLI tool designed primarily for **GitHub Actions** to man
 🎪 abc123f 🌐 52-1-2-3      # Available at http://52.1.2.3:8080
 ```
 
-## 📊 For Maintainers (CLI Operations)
+### 🔄 Showtime Workflow
+
+```mermaid
+flowchart TD
+    A[User adds 🎪 ⚡ trigger-start] --> B[GitHub Actions: sync]
+    B --> C{Current state?}
+
+    C -->|No environment| D[🔒 Claim: Remove trigger + Set building]
+    C -->|Running + new SHA| E[🔒 Claim: Remove trigger + Set building]
+    C -->|Already building| F[❌ Exit: Another job active]
+    C -->|No triggers| G[❌ Exit: Nothing to do]
+
+    D --> H[📋 State: building]
+    E --> H
+    H --> I[🐳 Docker build]
+    I -->|Success| J[📋 State: built]
+    I -->|Fail| K[📋 State: failed]
+
+    J --> L[📋 State: deploying]
+    L --> M[☁️ AWS Deploy]
+    M -->|Success| N[📋 State: running]
+    M -->|Fail| O[📋 State: failed]
+
+    N --> P[🎪 Environment ready!]
+
+    Q[User adds 🎪 🛑 trigger-stop] --> R[🧹 Cleanup AWS + Remove labels]
+```
 
-> **Note**: CLI is mainly for debugging or developing Showtime itself. Primary interface is GitHub labels above.
 
 **Install CLI for debugging:**
 ```bash
@@ -164,129 +189,31 @@ You'll see:
 
 ### GitHub Actions Integration
 
-Showtime is designed to be called by Superset's GitHub Actions workflows:
-
-```yaml
-# .github/workflows/showtime.yml - Integrates with Superset's existing build workflows
-on:
-  pull_request_target:
-    types: [labeled, unlabeled, synchronize]
-
-jobs:
-  showtime-handler:
-    if: contains(github.event.label.name, '🎪')
-    steps:
-      - name: Install Showtime from PyPI
-        run: pip install superset-showtime
-
-      - name: Process circus triggers
-        run: python -m showtime handle-trigger ${{ github.event.pull_request.number }}
-```
-
-**Integration approach:**
-- **Coordinates with Superset builds** - Uses existing container build workflows
-- **Runs trusted code** (from PyPI, not PR code)
-- **Simple orchestration logic** (install CLI and run commands)
-- **Leverages existing infrastructure** - Same AWS resources and permissions
-
-## 🛠️ Installation & Setup
-
-### For Contributors (GitHub Labels Only)
-No installation needed! Just use GitHub labels to trigger environments.
-
-### For Maintainers (Manual CLI Operations)
+**🎯 Live Workflow**: [showtime-trigger.yml](https://github.com/apache/superset/actions/workflows/showtime-trigger.yml)
 
-**Install CLI for debugging/testing:**
-```bash
-pip install superset-showtime
-export GITHUB_TOKEN=your_personal_access_token
-```
+**How it works:**
+- Triggers on PR label changes, commits, and closures
+- Installs `superset-showtime` from PyPI (trusted code, not PR code)
+- Runs `showtime sync` to handle trigger processing and deployments
+- Supports manual testing via `workflow_dispatch` with specific SHA override
 
-**Manual operations:**
+**Commands used:**
 ```bash
-showtime list                    # Monitor all active environments
-showtime status 1234            # Debug specific environment
-showtime labels                 # Reference complete label system
+showtime sync PR_NUMBER --check-only    # Determine build_needed + target_sha
+showtime sync PR_NUMBER --sha SHA       # Execute atomic claim + build + deploy
 ```
 
-### For Repository Integration (GitHub Actions)
-
-**1. Install GitHub workflows:**
-Copy `workflows-reference/showtime-trigger.yml` and `workflows-reference/showtime-cleanup.yml` to Superset's `.github/workflows/`.
-
-**2. Configure secrets (already exist in Superset):**
-- `AWS_ACCESS_KEY_ID`
-- `AWS_SECRET_ACCESS_KEY`
-- `GITHUB_TOKEN`
-
-**3. Dependencies:**
-Showtime coordinates with Superset's existing build infrastructure - no additional setup needed.
-
-## 📊 CLI Reference (For Development/Debugging)
+## 🛠️ CLI Usage
 
-> **Primary Interface**: Use GitHub labels in PR interface. CLI is mainly for maintainers debugging or developing Showtime itself.
+The CLI is primarily used by GitHub Actions, but available for debugging and advanced users:
 
-### Debugging Commands
 ```bash
-showtime list                         # Monitor all environments
-showtime status 1234                  # Debug specific environment
-showtime labels                       # Complete label reference
-showtime test-lifecycle 1234          # Full workflow simulation
-```
-
-### Manual Operations (Advanced)
-```bash
-showtime start 1234                    # Manually create environment
-showtime start 1234 --sha abc123f     # Create environment (specific SHA)
-showtime stop 1234                    # Manually delete environment
-showtime sync 1234                    # Force sync to desired state
-showtime cleanup --respect-ttl        # Manual cleanup
+pip install superset-showtime
+export GITHUB_TOKEN=your_token
+showtime --help  # See all available commands
 ```
 
-### GitHub Actions Commands
-```bash
-showtime handle-trigger 1234          # Process trigger labels (called by GHA)
-showtime cleanup --older-than 48h     # Scheduled cleanup (called by GHA)
-```
 
-## 🎪 Benefits for Superset
-
-### For Contributors
-- **🎯 Simple workflow** - Just add/remove GitHub labels
-- **👀 Visual feedback** - See environment status in PR labels
-- **⚡ Automatic updates** - New commits update environments automatically
-- **🔧 Configuration testing** - Test config changes through code commits
-
-### For Maintainers
-- **📊 Complete visibility** - `showtime list` shows all environments
-- **🧹 Easy cleanup** - Automatic expired environment cleanup
-- **🔍 Better debugging** - Clear state in labels, comprehensive CLI
-- **💰 Cost savings** - No duplicate environments, proper cleanup
-
-### For Operations
-- **📝 Simpler workflows** - Replace complex GHA scripts with simple CLI calls
-- **🔒 Same security model** - No new permissions needed
-- **🎯 Deterministic** - Predictable AWS resource naming
-- **🚨 Monitoring ready** - 48h maximum lifetime, scheduled cleanup
-
-## 🏗️ Architecture
-
-### State Management
-All state lives in **GitHub labels** - no external databases needed:
-- **Trigger labels** (`🎪 trigger-*`) - Commands that get processed and removed
-- **State labels** (`🎪 🚦 *`) - Current environment status, managed by CLI
-
-### AWS Resources
-Deterministic naming enables reliable cleanup:
-- **ECS Service:** `pr-{pr_number}-{sha}` (e.g., `pr-1234-abc123f`)
-- **ECR Image:** `pr-{pr_number}-{sha}-ci` (e.g., `pr-1234-abc123f-ci`)
-
-### Rolling Updates
-Zero-downtime updates by running multiple environments:
-1. Keep old environment serving traffic
-2. Build new environment in parallel
-3. Switch traffic when new environment is healthy
-4. Clean up old environment
 
 ## 🤝 Contributing
 

{superset_showtime-0.2.8 → superset_showtime-0.3.1}/showtime/__init__.py

@@ -4,7 +4,7 @@
 Circus tent emoji state tracking for Apache Superset ephemeral environments.
 """
 
-__version__ = "0.2.8"
+__version__ = "0.3.1"
 __author__ = "Maxime Beauchemin"
 __email__ = "maximebeauchemin@gmail.com"