container-superposition 0.1.1 → 0.1.3

package/overlays/duckdb/README.md

@@ -0,0 +1,274 @@
+# DuckDB Overlay
+
+In-process analytical database for OLAP workloads and data analysis.
+
+## Features
+
+- **DuckDB CLI** - Command-line interface for interactive queries
+- **In-process** - No separate server required, embedded database
+- **OLAP optimized** - Columnar storage for analytical queries
+- **SQL support** - Full ANSI SQL with extensions
+- **Parquet/CSV support** - Direct querying of file formats
+- **Python integration** - Works seamlessly with Python and pandas
+
+## How It Works
+
+This overlay installs the DuckDB CLI, an in-process analytical database designed for OLAP workloads. Unlike traditional databases, DuckDB runs embedded in your application without requiring a separate server.
+
+**Suggested overlays:**
+
+- `python` - For DuckDB Python API
+- `jupyter` - Interactive data analysis
+
+## Installation
+
+DuckDB CLI is installed automatically during devcontainer creation via `setup.sh`:
+
+- Downloads DuckDB CLI for your architecture (amd64/aarch64)
+- Installs to `/usr/local/bin/duckdb`
+- Verifies installation with test query
+
+## Common Commands
+
+### Interactive Shell
+
+```bash
+# Start DuckDB with in-memory database
+duckdb
+
+# Create/open persistent database
+duckdb mydata.db
+
+# Open database read-only
+duckdb mydata.db -readonly
+```
+
+### Query Files
+
+```bash
+# Query CSV file
+duckdb -c "SELECT * FROM 'data.csv' LIMIT 10"
+
+# Query Parquet file
+duckdb -c "SELECT * FROM 'data.parquet' WHERE col > 100"
+
+# Query JSON file
+duckdb -c "SELECT * FROM read_json_auto('data.json')"
+```
+
+### Create Tables
+
+```bash
+# From CSV
+CREATE TABLE sales AS SELECT * FROM 'sales.csv';
+
+# From Parquet
+CREATE TABLE events AS SELECT * FROM 'events.parquet';
+
+# From query
+CREATE TABLE summary AS
+  SELECT date, SUM(amount) as total
+  FROM sales
+  GROUP BY date;
+```
+
+### Export Results
+
+```bash
+# Export to CSV
+COPY (SELECT * FROM sales) TO 'output.csv' (HEADER, DELIMITER ',');
+
+# Export to Parquet
+COPY sales TO 'output.parquet' (FORMAT PARQUET);
+
+# Export to JSON
+COPY (SELECT * FROM sales LIMIT 100) TO 'sample.json';
+```
+
+## Python Integration
+
+Install DuckDB Python package:
+
+```python
+# In Python code or Jupyter notebook
+!pip install duckdb
+
+import duckdb
+
+# Connect to database
+con = duckdb.connect('mydata.db')
+
+# Execute query
+result = con.execute("SELECT * FROM sales WHERE amount > 100").fetchall()
+
+# Query pandas DataFrame directly
+import pandas as pd
+df = pd.read_csv('data.csv')
+result = con.execute("SELECT * FROM df WHERE col > 100").df()
+
+# Close connection
+con.close()
+```
+
+### Pandas Integration
+
+```python
+import duckdb
+import pandas as pd
+
+# Read CSV into DuckDB, query, get pandas DataFrame
+df = duckdb.query("SELECT * FROM 'data.csv' WHERE col > 100").df()
+
+# Query DataFrame with SQL
+df2 = duckdb.query("SELECT col1, AVG(col2) FROM df GROUP BY col1").df()
+```
+
+## Use Cases
+
+- **Data analysis** - Ad-hoc analysis of large datasets
+- **ETL pipelines** - Transform data with SQL
+- **Analytics** - OLAP queries on columnar data
+- **Data science** - Integrate with Python/Jupyter workflows
+- **Data exploration** - Quick insights from CSV/Parquet files
+- **Embedded analytics** - In-app analytical queries
+
+**Integrates well with:**
+
+- `python` - DuckDB Python API (suggested)
+- `jupyter` - Interactive data analysis (suggested)
+- Node.js, .NET - DuckDB client libraries available
+
+## SQL Examples
+
+### Analytical Queries
+
+```sql
+-- Time series aggregation
+SELECT
+  date_trunc('day', timestamp) as day,
+  COUNT(*) as events,
+  AVG(duration) as avg_duration
+FROM events
+WHERE timestamp > '2024-01-01'
+GROUP BY day
+ORDER BY day;
+
+-- Window functions
+SELECT
+  user_id,
+  event_time,
+  event_type,
+  ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY event_time) as event_seq
+FROM user_events;
+
+-- Join multiple files
+SELECT
+  u.name,
+  COUNT(o.id) as order_count,
+  SUM(o.amount) as total_spent
+FROM 'users.csv' u
+JOIN 'orders.parquet' o ON u.id = o.user_id
+GROUP BY u.name
+ORDER BY total_spent DESC;
+```
+
+### File Formats
+
+```sql
+-- CSV with options
+SELECT * FROM read_csv_auto('data.csv', header=true, delim='|');
+
+-- Parquet with projection
+SELECT col1, col2 FROM 'data.parquet' WHERE col3 > 100;
+
+-- JSON with auto-detection
+SELECT * FROM read_json_auto('data.json');
+
+-- Excel files
+INSTALL spatial;
+LOAD spatial;
+SELECT * FROM ST_Read('data.xlsx');
+```
+
+## Benefits vs PostgreSQL
+
+| Feature               | DuckDB                | PostgreSQL              |
+| --------------------- | --------------------- | ----------------------- |
+| **Use Case**          | ✅ Analytics (OLAP)   | ✅ Transactions (OLTP)  |
+| **Setup**             | ✅ No server needed   | ⚠️ Separate service     |
+| **File Queries**      | ✅ Direct CSV/Parquet | ❌ Requires COPY        |
+| **Analytical Speed**  | ✅ Very fast          | ⚠️ Slower for analytics |
+| **Concurrent Writes** | ⚠️ Limited            | ✅ Excellent            |
+| **Data Size**         | ✅ GBs-TBs            | ✅ TBs+                 |
+
+**When to use DuckDB:**
+
+- Analytical queries on read-heavy data
+- Working with files (CSV, Parquet, JSON)
+- Embedded analytics in applications
+- Data science and exploration
+
+**When to use PostgreSQL:**
+
+- Transactional workloads (OLTP)
+- Concurrent writes from multiple clients
+- Need full ACID guarantees
+- Large-scale production databases
+
+## Troubleshooting
+
+### Installation Fails
+
+Check architecture is supported:
+
+```bash
+uname -m
+# Should be x86_64 or aarch64/arm64
+```
+
+### File Not Found
+
+Ensure file paths are correct:
+
+```bash
+# Use absolute paths or paths relative to working directory
+duckdb -c "SELECT * FROM '/absolute/path/data.csv'"
+duckdb -c "SELECT * FROM './relative/path/data.csv'"
+```
+
+### Out of Memory
+
+DuckDB uses memory-mapped files. For large datasets:
+
+```sql
+-- Limit memory usage
+SET memory_limit='4GB';
+
+-- Use temp directory
+SET temp_directory='/tmp';
+```
+
+### Slow Queries
+
+DuckDB is optimized for analytics but:
+
+```sql
+-- Create indexes for point queries
+CREATE INDEX idx_id ON sales(id);
+
+-- Use columnar storage
+COPY sales TO 'sales.parquet' (FORMAT PARQUET);
+```
+
+## References
+
+- [DuckDB Documentation](https://duckdb.org/docs/)
+- [DuckDB Python API](https://duckdb.org/docs/api/python/overview)
+- [DuckDB SQL Reference](https://duckdb.org/docs/sql/introduction)
+- [DuckDB File Formats](https://duckdb.org/docs/data/overview)
+
+**Related Overlays:**
+
+- `python` - DuckDB Python API (suggested)
+- `jupyter` - Interactive analysis (suggested)
+- `postgres` - Transactional database

package/overlays/duckdb/devcontainer.patch.json

@@ -0,0 +1,10 @@
+{
+    "$schema": "https://raw.githubusercontent.com/devcontainers/spec/main/schemas/devContainer.base.schema.json",
+    "features": {
+        "./features/cross-distro-packages": {
+            "apt": "wget unzip",
+            "apk": "wget unzip"
+        }
+    },
+    "postCreateCommand": "bash .devcontainer/scripts/setup-duckdb.sh"
+}

package/overlays/duckdb/overlay.yml

@@ -0,0 +1,17 @@
+id: duckdb
+name: DuckDB
+description: In-process analytical database for OLAP workloads
+category: database
+supports: []
+requires: []
+suggests:
+    - python
+    - jupyter
+conflicts: []
+tags:
+    - database
+    - analytics
+    - sql
+    - duckdb
+    - olap
+ports: []

package/overlays/duckdb/setup.sh

@@ -0,0 +1,45 @@
+#!/bin/bash
+# Setup script for DuckDB
+
+set -e
+
+echo "🔧 Setting up DuckDB..."
+
+# Detect architecture
+ARCH=$(uname -m)
+case $ARCH in
+    x86_64)
+        DUCKDB_ARCH="amd64"
+        ;;
+    aarch64|arm64)
+        DUCKDB_ARCH="aarch64"
+        ;;
+    *)
+        echo "❌ Unsupported architecture: $ARCH"
+        exit 1
+        ;;
+esac
+
+# Install DuckDB CLI
+DUCKDB_VERSION="${DUCKDB_VERSION:-v1.0.0}"
+echo "📦 Installing DuckDB CLI ${DUCKDB_VERSION}..."
+
+wget -q "https://github.com/duckdb/duckdb/releases/download/${DUCKDB_VERSION}/duckdb_cli-linux-${DUCKDB_ARCH}.zip" -O /tmp/duckdb.zip
+unzip -q /tmp/duckdb.zip -d /tmp/
+chmod +x /tmp/duckdb
+sudo mv /tmp/duckdb /usr/local/bin/duckdb
+rm /tmp/duckdb.zip
+
+# Verify installation
+if command -v duckdb &> /dev/null; then
+    echo "✅ DuckDB CLI installed successfully"
+    duckdb --version
+else
+    echo "❌ DuckDB CLI installation failed"
+    exit 1
+fi
+
+echo "✅ DuckDB setup complete"
+echo ""
+echo "ℹ️  To start DuckDB, run:"
+echo "   duckdb mydata.db"

package/overlays/duckdb/verify.sh

@@ -0,0 +1,32 @@
+#!/bin/bash
+# Verification script for DuckDB overlay
+# Confirms DuckDB is installed
+
+set -e
+
+echo "🔍 Verifying DuckDB overlay..."
+echo ""
+
+# Check DuckDB CLI is installed
+echo "1️⃣ Checking DuckDB CLI installation..."
+if command -v duckdb &> /dev/null; then
+    duckdb --version
+    echo "   ✅ DuckDB CLI is installed"
+else
+    echo "   ❌ DuckDB CLI is not installed"
+    exit 1
+fi
+
+# Test DuckDB with a simple query
+echo ""
+echo "2️⃣ Testing DuckDB with simple query..."
+RESULT=$(echo "SELECT 'DuckDB is working!' as message;" | duckdb 2>&1 | grep -i "DuckDB is working" || true)
+if [ -n "$RESULT" ]; then
+    echo "   ✅ DuckDB query executed successfully"
+else
+    echo "   ❌ DuckDB query failed"
+    exit 1
+fi
+
+echo ""
+echo "✅ DuckDB overlay verification complete"

package/overlays/git-helpers/overlay.yml

@@ -13,3 +13,4 @@ tags:
     - ssh
     - gpg
 ports: []
+minimal: true

package/overlays/grafana/README.md

@@ -300,7 +300,7 @@ Access at: http://localhost:3000/dashboards
 2. Click **Dashboard settings** (gear icon)
 3. Click **JSON Model**
 4. Copy JSON
-5. Save to `.devcontainer/dashboards/my-dashboard.json`
+5. Save to `.devcontainer/dashboards-grafana/my-dashboard.json`
 6. Restart Grafana to load
 
 **Option 2: Export from Grafana.com**
@@ -308,14 +308,14 @@ Access at: http://localhost:3000/dashboards
 1. Browse https://grafana.com/grafana/dashboards/
 2. Find dashboard (e.g., Node Exporter Full)
 3. Download JSON
-4. Save to `.devcontainer/dashboards/`
+4. Save to `.devcontainer/dashboards-grafana/`
 5. Restart Grafana
 
 **Dashboard structure:**
 
 ```
 .devcontainer/
-├── dashboards/
+├── dashboards-grafana/
 │   ├── observability-overview.json
 │   ├── my-custom-dashboard.json
 │   └── node-exporter.json
@@ -376,13 +376,13 @@ docker exec tempo cat /etc/tempo/tempo-config.yaml
 docker exec grafana ls /etc/grafana/provisioning/dashboards/
 
 # Check dashboard files are present
-docker exec grafana ls /etc/grafana/provisioning/dashboards/*.json
+docker exec grafana ls /etc/grafana/dashboards/*.json
 
 # Check Grafana logs for errors
 docker logs grafana | grep -i dashboard
 
 # Validate JSON syntax
-jq . .devcontainer/dashboards/my-dashboard.json
+jq . .devcontainer/dashboards-grafana/my-dashboard.json
 ```
 
 ### Issue: Correlation Links Not Working

package/overlays/grafana/dashboard-provider.yml

@@ -8,4 +8,4 @@ providers:
       disableDeletion: false
       editable: true
       options:
-          path: /etc/grafana/provisioning/dashboards
+          path: /etc/grafana/dashboards

package/overlays/grafana/docker-compose.yml

@@ -8,8 +8,8 @@ services:
             - GF_USERS_ALLOW_SIGN_UP=false
         volumes:
             - ./grafana-datasources-grafana.yml:/etc/grafana/provisioning/datasources/datasources.yml
-            - ./dashboard-provider-grafana.yml:/etc/grafana/provisioning/dashboards/dashboards.yml
-            - ./dashboards:/etc/grafana/provisioning/dashboards
+            - ./dashboard-provider-grafana.yml:/etc/grafana/provisioning/dashboards/provider.yml
+            - ./dashboards-grafana:/etc/grafana/dashboards
             - grafana_data:/var/lib/grafana
         ports:
             - '${GRAFANA_PORT:-3000}:3000'

package/overlays/grafana/overlay.yml

@@ -17,5 +17,10 @@ tags:
     - ui
     - visualization
 ports:
-    - 3000
+    - port: 3000
+      service: grafana
+      protocol: http
+      description: Grafana web UI
+      path: /
+      onAutoForward: openBrowser
 order: 3

package/overlays/jaeger/overlay.yml

@@ -13,7 +13,20 @@ tags:
     - tracing
     - jaeger
 ports:
-    - 16686
-    - 14250
-    - 14268
+    - port: 16686
+      service: jaeger
+      protocol: http
+      description: Jaeger UI
+      path: /
+      onAutoForward: openBrowser
+    - port: 14250
+      service: jaeger
+      protocol: grpc
+      description: Jaeger gRPC receiver
+      onAutoForward: ignore
+    - port: 14268
+      service: jaeger
+      protocol: http
+      description: Jaeger HTTP receiver
+      onAutoForward: ignore
 order: 1

package/overlays/jupyter/.env.example

@@ -0,0 +1,6 @@
+# Jupyter Configuration
+JUPYTER_VERSION=latest
+JUPYTER_ENABLE_LAB=yes
+JUPYTER_TOKEN=
+JUPYTER_PORT=8888
+JUPYTER_NOTEBOOKS_PATH=./notebooks

package/overlays/jupyter/README.md

@@ -0,0 +1,210 @@
+# Jupyter Overlay
+
+Jupyter notebook server for interactive computing and data science.
+
+## Features
+
+- **Jupyter Notebook/Lab** - Interactive computing environment
+- **Python kernel** - Run Python code interactively
+- **Persistent notebooks** - Work saved across container restarts
+- **Volume mounting** - Access notebooks from host filesystem
+- **No authentication** - Pre-configured for development use
+- **Docker Compose service** - Runs as separate container
+
+## How It Works
+
+This overlay adds Jupyter as a Docker Compose service running in its own container. Jupyter is accessible from your development container and your browser via port 8888.
+
+**Dependencies:**
+
+- `python` (required) - Python runtime for notebooks
+
+**Architecture:**
+
+- Jupyter service runs on port 8888
+- Notebooks stored in Docker volume (`jupyter-data`)
+- Optional local directory mounting for notebooks
+- JupyterLab interface enabled by default
+
+## Configuration
+
+### Environment Variables
+
+The overlay includes a `.env.example` file. Copy it to `.env` and customize:
+
+```bash
+cd .devcontainer
+cp .env.example .env
+```
+
+**Default values (.env.example):**
+
+```bash
+# Jupyter Configuration
+JUPYTER_VERSION=latest
+JUPYTER_ENABLE_LAB=yes
+JUPYTER_TOKEN=
+JUPYTER_PORT=8888
+JUPYTER_NOTEBOOKS_PATH=./notebooks
+```
+
+### Notebook Directory
+
+By default, notebooks can be created in the Jupyter container. To use notebooks from your host:
+
+1. Create a `notebooks` directory in your project root
+2. Notebooks in this directory will be available at `/home/jovyan/notebooks` in Jupyter
+
+### Authentication
+
+By default, authentication is disabled for local development:
+
+- No token required
+- No password required
+
+For production or shared environments, set `JUPYTER_TOKEN`:
+
+```bash
+JUPYTER_TOKEN=your-secure-token
+```
+
+### Port Configuration
+
+The default port (8888) can be changed via the `--port-offset` option:
+
+```bash
+npm run init -- --port-offset 100 --stack compose --language python,jupyter
+# Jupyter will be on port 8988
+```
+
+## Common Commands
+
+### Accessing Jupyter
+
+```bash
+# Open in browser
+http://localhost:8888
+
+# Or use container hostname from dev container
+curl http://jupyter:8888
+```
+
+### Notebook Management
+
+Jupyter provides a web interface for all notebook operations:
+
+- Create new notebooks
+- Upload existing notebooks
+- Download notebooks
+- Organize in folders
+- Run code cells
+- Export to various formats (HTML, PDF, etc.)
+
+### Using Python in Notebooks
+
+```python
+# Install packages
+!pip install pandas numpy matplotlib seaborn
+
+# Import and use
+import pandas as pd
+import numpy as np
+import matplotlib.pyplot as plt
+
+# Your analysis code
+df = pd.read_csv('data.csv')
+df.head()
+```
+
+### JupyterLab Features
+
+JupyterLab is enabled by default and provides:
+
+- Multiple tabs
+- File browser
+- Terminal
+- Text editor
+- Markdown preview
+- Extension manager
+
+## Use Cases
+
+- **Data analysis** - Explore and visualize datasets
+- **Machine learning** - Develop and train models
+- **Scientific computing** - Mathematical and statistical analysis
+- **Documentation** - Create executable documentation
+- **Teaching** - Interactive tutorials and examples
+- **Prototyping** - Quick experimentation with code
+
+**Integrates well with:**
+
+- `python` - Python runtime (required)
+- `postgres` - Database for data analysis
+- `duckdb` - Analytics database
+- `redis` - Caching and data structures
+
+## Available Images
+
+Jupyter provides different base images:
+
+- `jupyter/minimal-notebook` - Minimal Jupyter with Python (default)
+- `jupyter/scipy-notebook` - Scientific Python stack
+- `jupyter/datascience-notebook` - Julia, Python, R
+- `jupyter/tensorflow-notebook` - TensorFlow and Keras
+- `jupyter/pyspark-notebook` - PySpark
+
+Change via `JUPYTER_VERSION`:
+
+```bash
+JUPYTER_VERSION=scipy-notebook
+```
+
+## Troubleshooting
+
+### Jupyter Not Starting
+
+Check service logs:
+
+```bash
+docker-compose logs jupyter
+```
+
+### Port Already in Use
+
+Change the port in `.env`:
+
+```bash
+JUPYTER_PORT=8889
+```
+
+Or use port offset when generating.
+
+### Notebooks Not Persisting
+
+Ensure you're saving notebooks to the mounted volume or directory:
+
+- `/home/jovyan/work` - Persistent volume
+- `/home/jovyan/notebooks` - Mounted from host
+
+### Package Installation
+
+Packages installed with `!pip install` are lost on container restart. To persist:
+
+1. Create a `requirements.txt` in your notebooks directory
+2. Install in a notebook: `!pip install -r /home/jovyan/notebooks/requirements.txt`
+
+Or extend the Jupyter image with a custom Dockerfile.
+
+## References
+
+- [Jupyter Documentation](https://jupyter.org/documentation)
+- [JupyterLab Documentation](https://jupyterlab.readthedocs.io/)
+- [Jupyter Docker Stacks](https://jupyter-docker-stacks.readthedocs.io/)
+- [Jupyter Notebook Basics](https://jupyter-notebook.readthedocs.io/en/stable/notebook.html)
+
+**Related Overlays:**
+
+- `python` - Python runtime (required)
+- `postgres` - Database for analysis
+- `duckdb` - Analytics database
+- `redis` - Data structures