container-superposition 0.1.1 → 0.1.3

package/overlays/.presets/data-engineering.yml

@@ -0,0 +1,392 @@
+# Data Engineering Preset
+# Python-based data engineering environment
+
+id: data-engineering
+name: Data Engineering
+description: Python-based data engineering with database, object storage, and analytics
+type: meta
+category: preset
+supports: [compose] # Requires Docker Compose for services
+tags: [preset, data-engineering, python, etl, analytics, data-science]
+
+# Overlays to select
+selects:
+    # Always included
+    required:
+        - python
+        - minio # Data lake storage
+        - modern-cli-tools
+
+    # User makes choices
+    userChoice:
+        database:
+            id: database
+            prompt: Select database for analytics
+            options: [postgres, mongodb]
+            defaultOption: postgres
+
+# Glue configuration - integration helpers
+glueConfig:
+    # Pre-configured environment variables
+    environment:
+        # Python environment
+        PYTHONUNBUFFERED: '1'
+        PYTHONDONTWRITEBYTECODE: '1'
+
+        # Database connections
+        # PostgreSQL
+        DATABASE_URL: 'postgresql://postgres:postgres@postgres:5432/datawarehouse'
+        POSTGRES_HOST: 'postgres'
+        POSTGRES_PORT: '5432'
+        POSTGRES_DB: 'datawarehouse'
+
+        # MongoDB (alternative)
+        MONGODB_URL: 'mongodb://mongodb:27017/datawarehouse'
+        MONGODB_HOST: 'mongodb'
+        MONGODB_PORT: '27017'
+        MONGODB_DB: 'datawarehouse'
+
+        # MinIO for data lake
+        MINIO_ENDPOINT: 'minio:9000'
+        MINIO_ACCESS_KEY: 'minioadmin'
+        MINIO_SECRET_KEY: 'minioadmin'
+        MINIO_BUCKET_RAW: 'raw-data'
+        MINIO_BUCKET_PROCESSED: 'processed-data'
+        MINIO_USE_SSL: 'false'
+
+        # Data processing settings
+        SPARK_MASTER: 'local[*]'
+        DASK_SCHEDULER: 'threads'
+
+        # Jupyter settings (for future jupyter overlay)
+        JUPYTER_PORT: '8888'
+        JUPYTER_TOKEN: 'data-engineering'
+
+    # Suggested port mappings
+    portMappings:
+        jupyter: 8888 # For future jupyter overlay
+        minio: 9000
+        minio-console: 9001
+
+    # README snippet to add to generated devcontainer
+    readme: |
+        ## Data Engineering Stack
+
+        This devcontainer is configured for data engineering and analytics:
+
+        ### Architecture
+
+        ```
+        Data Sources ──→ Raw Data (MinIO) ──→ ETL Pipeline (Python)
+                                                      │
+                                                      ↓
+                         Processed Data (MinIO) ──→ Analytics DB
+                                                      │
+                                                      ↓
+                                              Analysis/Reporting
+        ```
+
+        ### Services
+
+        - **Python**: Data processing runtime with scientific libraries
+        - **Database**: Analytics database (PostgreSQL or MongoDB)
+        - **MinIO**: S3-compatible data lake (ports 9000, 9001)
+        - **Modern CLI Tools**: jq, yq, bat, fd, ripgrep for data wrangling
+
+        ### Connection Strings
+
+        #### Database
+
+        ```bash
+        # PostgreSQL
+        DATABASE_URL="postgresql://postgres:postgres@postgres:5432/datawarehouse"
+
+        # MongoDB (alternative)
+        MONGODB_URL="mongodb://mongodb:27017/datawarehouse"
+        ```
+
+        #### MinIO (Data Lake)
+
+        ```bash
+        MINIO_ENDPOINT="minio:9000"
+        MINIO_ACCESS_KEY="minioadmin"
+        MINIO_SECRET_KEY="minioadmin"
+        MINIO_BUCKET_RAW="raw-data"
+        MINIO_BUCKET_PROCESSED="processed-data"
+        ```
+
+        ### Python Libraries
+
+        Pre-installed data science libraries are configured in `global-packages-python.txt`:
+
+        - **pandas**: Data manipulation and analysis
+        - **polars**: Fast DataFrame library (Rust-based)
+        - **numpy**: Numerical computing
+        - **sqlalchemy**: Database ORM
+        - **psycopg2**: PostgreSQL driver
+        - **pymongo**: MongoDB driver
+        - **boto3**: AWS SDK (for MinIO/S3)
+        - **requests**: HTTP library for API calls
+
+        Install additional libraries:
+
+        ```bash
+        pip install dask pyarrow fastparquet
+        ```
+
+        ### Quick Start
+
+        #### 1. Set Up MinIO Buckets
+
+        ```bash
+        # Access MinIO Console at http://localhost:9001
+        # Login: minioadmin/minioadmin
+
+        # Or use CLI:
+        mc alias set local http://minio:9000 minioadmin minioadmin
+        mc mb local/raw-data
+        mc mb local/processed-data
+        ```
+
+        #### 2. Upload Raw Data to MinIO
+
+        ```python
+        import boto3
+        from io import BytesIO
+
+        # Configure S3 client for MinIO
+        s3 = boto3.client(
+            's3',
+            endpoint_url='http://minio:9000',
+            aws_access_key_id='minioadmin',
+            aws_secret_access_key='minioadmin'
+        )
+
+        # Upload CSV file
+        s3.upload_file('data.csv', 'raw-data', 'sales/2024/data.csv')
+
+        # Or upload from memory
+        csv_buffer = BytesIO()
+        df.to_csv(csv_buffer, index=False)
+        s3.put_object(
+            Bucket='raw-data',
+            Key='sales/2024/data.csv',
+            Body=csv_buffer.getvalue()
+        )
+        ```
+
+        #### 3. ETL Pipeline Example
+
+        ```python
+        import pandas as pd
+        import boto3
+        from sqlalchemy import create_engine
+
+        # Initialize connections
+        s3 = boto3.client('s3', endpoint_url='http://minio:9000',
+                         aws_access_key_id='minioadmin',
+                         aws_secret_access_key='minioadmin')
+        engine = create_engine('postgresql://postgres:postgres@postgres:5432/datawarehouse')
+
+        # Extract: Read from MinIO
+        obj = s3.get_object(Bucket='raw-data', Key='sales/2024/data.csv')
+        df = pd.read_csv(obj['Body'])
+
+        # Transform: Clean and process
+        df = df.dropna()
+        df['date'] = pd.to_datetime(df['date'])
+        df['revenue'] = df['quantity'] * df['price']
+
+        # Load: Save to database
+        df.to_sql('sales', engine, if_exists='append', index=False)
+
+        # Archive processed data to MinIO
+        df.to_parquet('/tmp/processed.parquet')
+        s3.upload_file('/tmp/processed.parquet', 'processed-data', 
+                      'sales/2024/processed.parquet')
+        ```
+
+        ### Data Processing Patterns
+
+        #### Batch Processing with Pandas
+
+        ```python
+        import pandas as pd
+
+        # Read data in chunks for large files
+        chunk_size = 10000
+        chunks = pd.read_csv('large_file.csv', chunksize=chunk_size)
+
+        for chunk in chunks:
+            # Process chunk
+            processed = chunk.groupby('category').agg({
+                'revenue': 'sum',
+                'quantity': 'count'
+            })
+            # Write to database
+            processed.to_sql('aggregates', engine, if_exists='append')
+        ```
+
+        #### Using Polars (Faster)
+
+        ```python
+        import polars as pl
+
+        # Read and process with Polars
+        df = pl.read_csv('data.csv')
+        result = (
+            df
+            .filter(pl.col('revenue') > 1000)
+            .group_by('category')
+            .agg([
+                pl.col('revenue').sum().alias('total_revenue'),
+                pl.col('quantity').mean().alias('avg_quantity')
+            ])
+        )
+        ```
+
+        #### Data Lake Organization
+
+        Organize data in MinIO using partitions:
+
+        ```
+        raw-data/
+        ├── sales/
+        │   ├── 2024/
+        │   │   ├── 01/
+        │   │   │   └── data.csv
+        │   │   └── 02/
+        │   └── 2023/
+        └── inventory/
+            └── current/
+                └── stock.json
+
+        processed-data/
+        ├── sales/
+        │   ├── daily_aggregates/
+        │   │   └── 2024-01-15.parquet
+        │   └── monthly_summary/
+        │       └── 2024-01.parquet
+        ```
+
+        ### Database Analytics
+
+        #### PostgreSQL
+
+        ```python
+        import pandas as pd
+        from sqlalchemy import create_engine
+
+        engine = create_engine('postgresql://postgres:postgres@postgres:5432/datawarehouse')
+
+        # Read data
+        df = pd.read_sql('SELECT * FROM sales WHERE date >= CURRENT_DATE - 30', engine)
+
+        # Write data
+        df.to_sql('sales_summary', engine, if_exists='replace')
+
+        # Execute custom SQL
+        with engine.connect() as conn:
+            result = conn.execute("""
+                SELECT category, SUM(revenue) as total
+                FROM sales
+                GROUP BY category
+                ORDER BY total DESC
+            """)
+            for row in result:
+                print(row)
+        ```
+
+        ### Modern CLI Tools for Data
+
+        ```bash
+        # Inspect JSON data
+        cat data.json | jq '.[] | select(.revenue > 1000)'
+
+        # Search in CSV files
+        rg -t csv 'product_name'
+
+        # Find data files
+        fd -e csv -e parquet -e json
+
+        # Preview files with syntax highlighting
+        bat sales.csv
+        ```
+
+        ### Workflow Automation
+
+        Create data pipelines with Python scripts:
+
+        ```python
+        # pipeline.py
+        import schedule
+        import time
+
+        def etl_job():
+            print("Running ETL pipeline...")
+            # Extract, transform, load
+            pass
+
+        # Run every hour
+        schedule.every().hour.do(etl_job)
+
+        while True:
+            schedule.run_pending()
+            time.sleep(60)
+        ```
+
+        Or use cron:
+
+        ```bash
+        # Add to crontab
+        0 * * * * python /workspace/pipeline.py
+        ```
+
+        ### Data Quality
+
+        ```python
+        # Data validation with pandas
+        def validate_data(df):
+            # Check for nulls
+            assert df['revenue'].notna().all(), "Revenue has null values"
+
+            # Check data types
+            assert df['date'].dtype == 'datetime64[ns]', "Date is not datetime"
+
+            # Check ranges
+            assert (df['quantity'] > 0).all(), "Quantity must be positive"
+
+            # Check uniqueness
+            assert df['transaction_id'].is_unique, "Duplicate transactions found"
+
+        validate_data(df)
+        ```
+
+        ### Future Enhancements
+
+        When Jupyter and DuckDB overlays are available:
+
+        #### Jupyter Notebooks
+
+        - Interactive data exploration
+        - Visualization with matplotlib, seaborn
+        - Experiment tracking
+        - Collaborative analysis
+
+        #### DuckDB
+
+        - In-process SQL analytics
+        - Query MinIO Parquet files directly
+        - Faster than loading to pandas
+        - OLAP-style aggregations
+
+        ### Next Steps
+
+        - Design your data lake structure in MinIO
+        - Set up database schemas for analytics
+        - Create ETL pipelines for data ingestion
+        - Implement data quality checks
+        - Set up scheduled jobs for recurring pipelines
+        - Add data monitoring and alerting
+        - Document data lineage
+        - Create data catalog

package/overlays/.presets/event-sourced-service.yml

@@ -0,0 +1,262 @@
+# Event-Sourced Service Preset
+# Development environment for event-driven/CQRS applications
+
+id: event-sourced-service
+name: Event-Sourced Service
+description: Event-driven architecture with event store, message broker, and observability
+type: meta
+category: preset
+supports: [compose] # Requires Docker Compose for services
+tags: [preset, event-sourcing, cqrs, messaging, event-driven]
+
+# Overlays to select
+selects:
+    # Always included
+    required:
+        - minio # Object storage for events
+        - otel-collector
+        - jaeger
+        - prometheus
+        - grafana
+
+    # User makes choices
+    userChoice:
+        language:
+            id: language
+            prompt: Select service language/framework
+            options: [nodejs, dotnet, python, go, java]
+            defaultOption: nodejs
+
+        eventStore:
+            id: eventStore
+            prompt: Select event store database
+            options: [postgres, mongodb]
+            defaultOption: postgres
+
+        messaging:
+            id: messaging
+            prompt: Select message broker
+            options: [rabbitmq, redpanda, nats]
+            defaultOption: rabbitmq
+
+# Glue configuration - integration helpers
+glueConfig:
+    # Pre-configured environment variables
+    environment:
+        # Event store configuration (varies by choice)
+        # PostgreSQL event store
+        EVENT_STORE_URL: 'postgresql://postgres:postgres@postgres:5432/eventstore'
+        POSTGRES_HOST: 'postgres'
+        POSTGRES_PORT: '5432'
+        POSTGRES_DB: 'eventstore'
+
+        # MongoDB event store (alternative)
+        MONGODB_URL: 'mongodb://mongodb:27017/eventstore'
+        MONGODB_HOST: 'mongodb'
+        MONGODB_PORT: '27017'
+        MONGODB_DB: 'eventstore'
+
+        # MinIO for event snapshots and archives
+        MINIO_ENDPOINT: 'minio:9000'
+        MINIO_ACCESS_KEY: 'minioadmin'
+        MINIO_SECRET_KEY: 'minioadmin'
+        MINIO_BUCKET: 'events'
+        MINIO_USE_SSL: 'false'
+
+        # Messaging URLs (varies by choice)
+        # RabbitMQ
+        RABBITMQ_URL: 'amqp://rabbitmq:5672'
+
+        # Redpanda (Kafka-compatible)
+        KAFKA_BROKERS: 'redpanda:9092'
+
+        # NATS
+        NATS_URL: 'nats://nats:4222'
+
+        # OpenTelemetry configuration
+        OTEL_EXPORTER_OTLP_ENDPOINT: 'http://otel-collector:4317'
+        OTEL_SERVICE_NAME: 'event-service'
+        OTEL_METRICS_EXPORTER: 'otlp'
+        OTEL_TRACES_EXPORTER: 'otlp'
+        OTEL_LOGS_EXPORTER: 'otlp'
+
+        # Event sourcing settings
+        EVENT_STORE_SNAPSHOT_THRESHOLD: '100'
+        EVENT_STORE_RETENTION_DAYS: '30'
+
+    # Suggested port mappings
+    portMappings:
+        service: 8080
+        grafana: 3000
+        prometheus: 9090
+        jaeger: 16686
+        minio: 9000
+        minio-console: 9001
+
+    # README snippet to add to generated devcontainer
+    readme: |
+        ## Event-Sourced Service Stack
+
+        This devcontainer is configured for event-driven architecture development:
+
+        ### Architecture Pattern
+
+        ```
+        Commands ──→ Service ──→ Event Store (append-only)
+                        │              │
+                        ↓              ↓
+                  Message Broker ← Events Published
+                        │
+                        ↓
+          Read Models / Projections
+        ```
+
+        ### Services
+
+        - **Event Store**: Your chosen database (PostgreSQL or MongoDB) for event persistence
+        - **Message Broker**: Async event distribution (RabbitMQ, Redpanda, or NATS)
+        - **MinIO**: Object storage for event snapshots and archives (ports 9000, 9001)
+        - **OpenTelemetry Collector**: Distributed tracing (ports 4317, 4318)
+        - **Jaeger**: Trace visualization (port 16686)
+        - **Prometheus**: Metrics storage (port 9090)
+        - **Grafana**: Observability dashboard (port 3000)
+
+        ### Connection Strings
+
+        #### Event Store
+
+        ```bash
+        # PostgreSQL
+        EVENT_STORE_URL="postgresql://postgres:postgres@postgres:5432/eventstore"
+
+        # MongoDB (alternative)
+        MONGODB_URL="mongodb://mongodb:27017/eventstore"
+        ```
+
+        #### Message Broker
+
+        ```bash
+        # RabbitMQ
+        RABBITMQ_URL="amqp://rabbitmq:5672"
+
+        # Redpanda (Kafka-compatible)
+        KAFKA_BROKERS="redpanda:9092"
+
+        # NATS
+        NATS_URL="nats://nats:4222"
+        ```
+
+        #### MinIO (Event Snapshots)
+
+        ```bash
+        MINIO_ENDPOINT="minio:9000"
+        MINIO_ACCESS_KEY="minioadmin"
+        MINIO_SECRET_KEY="minioadmin"
+        MINIO_BUCKET="events"
+        ```
+
+        ### Quick Start
+
+        1. **Start your service** on port 8080
+
+        2. **Set up event store schema**:
+           ```bash
+           # PostgreSQL example - create events table
+           psql -h postgres -U postgres -d eventstore <<EOF
+           CREATE TABLE events (
+               event_id UUID PRIMARY KEY,
+               aggregate_id UUID NOT NULL,
+               aggregate_type VARCHAR(255) NOT NULL,
+               event_type VARCHAR(255) NOT NULL,
+               event_data JSONB NOT NULL,
+               metadata JSONB,
+               version BIGINT NOT NULL,
+               timestamp TIMESTAMP DEFAULT NOW(),
+               UNIQUE(aggregate_id, version)
+           );
+           CREATE INDEX idx_aggregate ON events(aggregate_id);
+           CREATE INDEX idx_timestamp ON events(timestamp);
+           EOF
+           ```
+
+        3. **Configure MinIO bucket**:
+           - Access MinIO Console at http://localhost:9001
+           - Login with minioadmin/minioadmin
+           - Create bucket named "events"
+           - Configure lifecycle policies for event retention
+
+        4. **Set up message broker**:
+           - RabbitMQ: http://localhost:15672 (guest/guest)
+           - Configure exchanges and queues for event distribution
+
+        ### Event Sourcing Patterns
+
+        This stack supports common event sourcing patterns:
+
+        #### Command Handler → Event Store
+
+        ```typescript
+        // Append events to store
+        async function handleCommand(command: Command) {
+            const events = aggregate.process(command);
+            await eventStore.append(aggregateId, events);
+            await messageBroker.publish(events);
+        }
+        ```
+
+        #### Event Replay / Projection
+
+        ```typescript
+        // Rebuild read models from events
+        async function rebuildProjection() {
+            const events = await eventStore.getEvents(fromTimestamp);
+            for (const event of events) {
+                await projection.handle(event);
+            }
+        }
+        ```
+
+        #### Snapshotting
+
+        ```typescript
+        // Store snapshots in MinIO every N events
+        if (aggregate.version % SNAPSHOT_THRESHOLD === 0) {
+            const snapshot = aggregate.toSnapshot();
+            await minioClient.putObject('events', 
+                `snapshots/${aggregateId}/${version}.json`, 
+                JSON.stringify(snapshot)
+            );
+        }
+        ```
+
+        ### Observability
+
+        - **Traces**: View command flows in Jaeger (http://localhost:16686)
+        - **Metrics**: Monitor event throughput in Prometheus/Grafana
+        - **Logs**: Centralized logging via OTEL → Loki
+
+        ### Event Store Best Practices
+
+        1. **Append-only**: Never modify or delete events
+        2. **Versioning**: Use optimistic concurrency with version numbers
+        3. **Idempotency**: Handle duplicate events gracefully
+        4. **Snapshots**: Store aggregate snapshots every N events
+        5. **Retention**: Archive old events to MinIO for compliance
+
+        ### Message Broker Patterns
+
+        - **Publish/Subscribe**: Broadcast events to multiple consumers
+        - **Competing Consumers**: Scale event handlers horizontally
+        - **Dead Letter Queue**: Handle failed event processing
+        - **Event Ordering**: Ensure ordered processing per aggregate
+
+        ### Next Steps
+
+        - Design your domain events and aggregates
+        - Implement command handlers with event sourcing
+        - Create read model projections
+        - Set up event subscribers on message broker
+        - Configure OTEL instrumentation for tracing
+        - Create Grafana dashboards for event metrics
+        - Implement snapshotting strategy
+        - Set up event archival to MinIO