@weirdfingers/baseboards 0.2.0

package/templates/api/docs/storage_providers.md

@@ -0,0 +1,337 @@
+# Storage Providers
+
+Boards supports multiple storage providers for artifact storage through a pluggable architecture. This document describes the available providers and their configuration options.
+
+## Available Providers
+
+### Local Storage
+
+The local filesystem provider is ideal for development and testing.
+
+**Features:**
+
+- Stores files on the local filesystem
+- Optional public URL serving for development
+- No external dependencies
+- Fast for small to medium files
+
+**Configuration:**
+
+```yaml
+providers:
+  local:
+    type: "local"
+    config:
+      base_path: "/tmp/boards/storage" # Storage directory
+      public_url_base: "http://localhost:8088/api/storage" # Optional: base URL for serving files
+```
+
+**Use Cases:**
+
+- Local development
+- Testing environments
+- Small deployments without cloud requirements
+
+---
+
+### AWS S3
+
+The S3 provider offers robust cloud storage with optional CloudFront CDN integration.
+
+**Features:**
+
+- Scalable cloud storage
+- Server-side encryption
+- CloudFront CDN integration
+- Presigned URLs for direct client uploads
+- Multiple storage classes for cost optimization
+- Cross-region replication support
+
+**Dependencies:**
+
+```bash
+pip install boto3 aioboto3
+# or
+pip install boards-backend[storage-s3]
+```
+
+**Configuration:**
+
+```yaml
+providers:
+  s3:
+    type: "s3"
+    config:
+      bucket: "my-boards-bucket" # Required: S3 bucket name
+      region: "us-east-1" # AWS region
+      aws_access_key_id: "${AWS_ACCESS_KEY_ID}" # AWS credentials
+      aws_secret_access_key: "${AWS_SECRET_ACCESS_KEY}"
+      aws_session_token: "${AWS_SESSION_TOKEN}" # Optional: for temporary credentials
+      endpoint_url: "https://s3.amazonaws.com" # Optional: custom endpoint (for S3-compatible services)
+      cloudfront_domain: "d123.cloudfront.net" # Optional: CloudFront distribution
+      upload_config: # Optional: S3 upload parameters
+        ServerSideEncryption: "AES256"
+        StorageClass: "STANDARD"
+        # Or use KMS encryption:
+        # ServerSideEncryption: "aws:kms"
+        # SSEKMSKeyId: "arn:aws:kms:region:account:key/key-id"
+```
+
+**Environment Variables:**
+
+- `AWS_ACCESS_KEY_ID`: AWS access key (if not in config)
+- `AWS_SECRET_ACCESS_KEY`: AWS secret key (if not in config)
+- `AWS_SESSION_TOKEN`: Temporary session token (optional)
+- `AWS_REGION`: Default region (if not in config)
+
+**Use Cases:**
+
+- Production deployments
+- Large file storage
+- Global content distribution with CloudFront
+- Compliance requirements (encryption, audit trails)
+- Integration with other AWS services
+
+---
+
+### Google Cloud Storage (GCS)
+
+The GCS provider integrates with Google Cloud Platform and Cloud CDN.
+
+**Features:**
+
+- Google Cloud native storage
+- Automatic encryption at rest
+- Cloud CDN integration
+- Multi-regional storage options
+- IAM integration with fine-grained permissions
+- Lifecycle management policies
+
+**Dependencies:**
+
+```bash
+pip install google-cloud-storage
+# or
+pip install boards-backend[storage-gcs]
+```
+
+**Configuration:**
+
+```yaml
+providers:
+  gcs:
+    type: "gcs"
+    config:
+      bucket: "my-boards-gcs-bucket" # Required: GCS bucket name
+      project_id: "my-gcp-project" # GCP project ID
+      # Authentication options (choose one):
+      credentials_path: "/path/to/service-account-key.json" # Service account key file
+      credentials_json: "${GCP_SERVICE_ACCOUNT_JSON}" # Service account JSON as string
+      # Or use default credentials (gcloud, environment, etc.)
+      cdn_domain: "cdn.example.com" # Optional: Cloud CDN domain
+      upload_config: # Optional: GCS upload parameters
+        cache_control: "public, max-age=3600"
+        predefined_acl: "bucket-owner-read"
+```
+
+**Authentication Methods:**
+
+1. **Service Account Key File:**
+
+   ```yaml
+   credentials_path: "/path/to/service-account-key.json"
+   ```
+
+2. **Service Account JSON String:**
+
+   ```yaml
+   credentials_json: "${GCP_SERVICE_ACCOUNT_JSON}"
+   ```
+
+3. **Default Credentials (recommended for production):**
+   - Google Cloud SDK (`gcloud auth application-default login`)
+   - Environment variable `GOOGLE_APPLICATION_CREDENTIALS`
+   - Compute Engine/GKE service accounts
+   - Cloud Run/Cloud Functions default credentials
+
+**Use Cases:**
+
+- Google Cloud Platform deployments
+- Integration with other GCP services
+- Global content distribution
+- Machine learning workflows (easy integration with AI Platform)
+- Multi-regional applications
+
+---
+
+### Supabase Storage
+
+The Supabase provider integrates with Supabase's storage and authentication system.
+
+**Features:**
+
+- Integrated with Supabase auth
+- Built-in CDN
+- Row Level Security (RLS) integration
+- Real-time subscriptions
+- Automatic image transformations
+
+**Dependencies:**
+
+```bash
+pip install supabase
+```
+
+**Configuration:**
+
+```yaml
+providers:
+  supabase:
+    type: "supabase"
+    config:
+      url: "${SUPABASE_URL}" # Supabase project URL
+      key: "${SUPABASE_ANON_KEY}" # Supabase anon key
+      bucket: "boards-artifacts" # Storage bucket name
+```
+
+**Use Cases:**
+
+- Supabase-based applications
+- Real-time collaborative features
+- Integrated authentication and storage
+- Rapid prototyping
+
+## Provider Selection and Routing
+
+Boards supports intelligent provider selection through routing rules:
+
+```yaml
+routing_rules:
+  # Large files go to S3
+  - condition:
+      size_gt: "50MB"
+    provider: "s3"
+
+  # Images go to GCS with CDN
+  - condition:
+      artifact_type: "image"
+    provider: "gcs"
+
+  # Videos go to S3 for CloudFront streaming
+  - condition:
+      artifact_type: "video"
+    provider: "s3"
+
+  # Default fallback
+  - provider: "local"
+```
+
+## Best Practices
+
+### Security
+
+1. **Never hardcode credentials** in configuration files
+2. **Use environment variables** or secure credential management
+3. **Enable encryption at rest** for sensitive data
+4. **Use IAM roles** instead of access keys when possible
+5. **Implement bucket policies** for additional security
+
+### Performance
+
+1. **Choose regions close to your users** for better latency
+2. **Use CDN/CloudFront** for frequently accessed content
+3. **Consider storage classes** for cost optimization
+4. **Implement caching** at the application level
+5. **Use presigned URLs** for direct client uploads
+
+### Cost Optimization
+
+1. **Use appropriate storage classes** (Standard, IA, Glacier, etc.)
+2. **Implement lifecycle policies** for automatic data management
+3. **Monitor usage and costs** regularly
+4. **Use compression** for text-based content
+5. **Clean up unused files** periodically
+
+### Development vs Production
+
+**Development:**
+
+```yaml
+default_provider: "local"
+providers:
+  local:
+    type: "local"
+    config:
+      base_path: "/tmp/boards/storage"
+```
+
+**Production:**
+
+```yaml
+default_provider: "s3"
+providers:
+  s3:
+    type: "s3"
+    config:
+      bucket: "prod-boards-artifacts"
+      cloudfront_domain: "cdn.example.com"
+  gcs:
+    type: "gcs"
+    config:
+      bucket: "prod-boards-media"
+      cdn_domain: "media.example.com"
+```
+
+## Migration Between Providers
+
+When migrating between providers, consider:
+
+1. **Gradual migration** using routing rules
+2. **Data consistency** during the migration
+3. **URL updates** for existing references
+4. **Testing** with non-critical data first
+5. **Backup strategies** for data safety
+
+Example gradual migration:
+
+```yaml
+# Phase 1: Route new uploads to S3, keep existing on local
+routing_rules:
+  - condition:
+      artifact_type: "image"
+    provider: "s3"
+  - provider: "local"  # Existing files
+
+# Phase 2: Migrate all new uploads to S3
+routing_rules:
+  - provider: "s3"
+```
+
+## Troubleshooting
+
+### Common Issues
+
+1. **Authentication errors**: Check credentials and permissions
+2. **Bucket access denied**: Verify bucket policies and IAM permissions
+3. **Network timeouts**: Check network connectivity and endpoint URLs
+4. **Large file uploads**: Consider chunked uploads and timeout settings
+5. **CDN cache issues**: Check cache headers and purge policies
+
+### Debugging
+
+Enable debug logging:
+
+```python
+import logging
+logging.getLogger("boards.storage").setLevel(logging.DEBUG)
+```
+
+### Monitoring
+
+Key metrics to monitor:
+
+- Upload success rates
+- Download latency
+- Storage costs
+- Error rates by provider
+- CDN hit rates (if applicable)

package/templates/api/pyproject.toml

@@ -0,0 +1,165 @@
+[build-system]
+requires = ["setuptools>=61.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "weirdfingers-boards"
+dynamic = ["version"]
+description = "Backend API server for Boards - AI creative toolkit with GraphQL, auth adapters, and generation workers"
+readme = "README.md"
+requires-python = ">=3.12"
+license = { text = "MIT" }
+authors = [
+    { name = "Boards Contributors", email = "ai.overlord@weirdfingers.com" },
+]
+keywords = [
+    "ai",
+    "creative",
+    "toolkit",
+    "graphql",
+    "fastapi",
+    "strawberry",
+    "auth",
+    "generation",
+]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: Internet :: WWW/HTTP :: HTTP Servers",
+    "Topic :: Software Development :: Libraries :: Application Frameworks",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+    "Framework :: FastAPI",
+    "Framework :: AsyncIO",
+]
+dependencies = [
+    "sqlalchemy>=2.0.0",
+    "supabase>=2.0.0",
+    "pydantic>=2.0.0",
+    "pydantic-settings>=2.0.0",
+    "httpx>=0.24.0",
+    "python-dotenv>=1.0.0",
+    "strawberry-graphql[fastapi]>=0.200.0",
+    "fastapi>=0.100.0",
+    "uvicorn[standard]>=0.23.0",
+    "dramatiq[redis,watch]>=1.15.0",
+    "pillow>=10.0.0",
+    "numpy>=1.24.0",
+    "psycopg2-binary>=2.9.0",
+    "structlog>=23.0.0",
+    "asyncpg>=0.29.0",
+    "alembic>=1.13.2",
+    "aiofiles>=23.0.0",
+    "pyyaml>=6.0.0",
+    "click>=8.0.0",
+    "redis>=5.0.0",
+    "pyjwt[crypto]>=2.8.0",
+    "psutil>=7.0.0",
+]
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=7.0.0",
+    "pytest-asyncio>=0.21.0",
+    "pytest-postgresql>=6.0.0",
+    "pytest-cov>=4.0.0",
+    "ruff>=0.1.0",
+    "pyright>=1.1.0",
+    "black>=23.0.0",
+    "pre-commit>=3.0.0",
+    # Include all provider deps for typecheck
+    "openai>=1.60.1",
+    "anthropic>=0.25.0",
+    "replicate>=1.0.4",
+    "together>=1.0.0",
+    # Include all storage deps for typecheck
+    "boto3>=1.34.0",
+    "aioboto3>=12.0.0",
+    "google-cloud-storage>=2.10.0",
+    # Type stubs
+    "types-redis>=4.6.0",
+]
+providers = [
+    "openai>=1.60.1",
+    "anthropic>=0.25.0",
+    "replicate>=1.0.4",
+    "together>=1.0.0",
+]
+storage-s3 = ["boto3>=1.34.0", "aioboto3>=12.0.0"]
+storage-gcs = ["google-cloud-storage>=2.10.0"]
+
+[project.urls]
+Homepage = "https://github.com/weirdfingers/boards"
+Documentation = "https://docs.boards.io"
+Repository = "https://github.com/weirdfingers/boards"
+Issues = "https://github.com/weirdfingers/boards/issues"
+Changelog = "https://github.com/weirdfingers/boards/blob/main/CHANGELOG.md"
+
+[project.scripts]
+boards-server = "boards.cli:main"
+boards-worker = "boards.workers.cli:main"
+boards-migrate = "boards.database.cli:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.ruff]
+line-length = 100
+target-version = "py312"
+exclude = ["alembic"]
+
+[tool.ruff.lint]
+select = [
+    "E",  # pycodestyle errors
+    "W",  # pycodestyle warnings
+    "F",  # pyflakes
+    "I",  # isort
+    "B",  # flake8-bugbear
+    "C4", # flake8-comprehensions
+    "UP", # pyupgrade
+]
+
+[tool.ruff.lint.per-file-ignores]
+# FastAPI uses Depends in function signatures
+"src/boards/api/**/*.py" = ["B008"]
+"tests/**/test_*.py" = ["B008"]
+# Alembic needs imports after path manipulation
+"alembic/env.py" = ["E402"]
+# __init__.py files often have unused imports for re-export
+"**/__init__.py" = ["F401"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+pythonpath = ["src"]
+asyncio_mode = "auto"
+
+[tool.setuptools.dynamic]
+version = { attr = "boards.__version__" }
+
+[tool.uv]
+dev-dependencies = [
+    "pytest>=7.0.0",
+    "pytest-asyncio>=0.21.0",
+    "pytest-postgresql>=6.0.0",
+    "pytest-cov>=4.0.0",
+    "ruff>=0.1.0",
+    "pyright>=1.1.0",
+    "black>=23.0.0",
+    "pre-commit>=3.0.0",
+    "build>=1.3.0",
+    "twine>=6.1.0",
+    "watchfiles>=1.1.0",
+    # Include all provider deps for typecheck
+    "openai>=1.60.1",
+    "anthropic>=0.25.0",
+    "replicate>=1.0.4",
+    "together>=1.0.0",
+    # Include all storage deps for typecheck
+    "boto3>=1.34.0",
+    "aioboto3>=12.0.0",
+    "google-cloud-storage>=2.10.0",
+    # Type stubs
+    "types-redis>=4.6.0",
+]

package/templates/api/src/boards/__init__.py

@@ -0,0 +1,10 @@
+"""
+Boards Backend SDK
+Open-source creative toolkit for AI-generated content
+"""
+
+__version__ = "0.2.0"
+
+from .config import settings
+
+__all__ = ["settings", "__version__"]

package/templates/api/src/boards/api/app.py

@@ -0,0 +1,171 @@
+"""
+Main FastAPI application for Boards backend
+"""
+
+import os
+from contextlib import asynccontextmanager
+
+from fastapi import FastAPI
+from fastapi.middleware.cors import CORSMiddleware
+
+from ..config import initialize_generator_api_keys, settings
+from ..database import init_database
+from ..generators.loader import load_generators_from_config
+from ..logging import configure_logging, get_logger
+from ..middleware import LoggingContextMiddleware, TenantRoutingMiddleware
+
+# Configure logging before creating logger
+configure_logging(debug=settings.debug)
+logger = get_logger(__name__)
+
+
+@asynccontextmanager
+async def lifespan(app: FastAPI):
+    """Application lifespan manager."""
+    # Startup
+    logger.info("Starting Boards API...")
+    init_database()
+    logger.info("Database initialized")
+
+    # Initialize generator API keys from settings
+    initialize_generator_api_keys()
+    logger.info("Generator API keys initialized")
+
+    # Load generators based on configuration
+    try:
+        load_generators_from_config()
+        logger.info(
+            "Generators configured and registered",
+            names=list(
+                __import__(
+                    "boards.generators.registry", fromlist=["registry"]
+                ).registry.list_names()
+            ),
+        )
+    except Exception as e:
+        # Fail fast: strict mode default may raise here
+        logger.error("Failed to configure generators", error=str(e))
+        raise
+
+    # Run comprehensive startup validation
+    try:
+        from ..validation import (
+            ValidationError,
+            get_startup_recommendations,
+            validate_startup_configuration,
+        )
+
+        validation_results = await validate_startup_configuration()
+
+        if not validation_results["overall_valid"]:
+            # Log detailed error information
+            logger.error(
+                "Application configuration validation failed - some features may not work properly",
+                database_errors=validation_results.get("database", {}).get("errors", []),
+                tenant_errors=validation_results["tenant"].get("errors", []),
+                auth_errors=validation_results["auth"].get("errors", []),
+            )
+
+            # In production, we might want to fail hard here
+            if settings.environment.lower() in ("production", "prod"):
+                raise ValidationError("Critical configuration validation failed in production")
+
+        # Log recommendations
+        recommendations = get_startup_recommendations(validation_results)
+        if recommendations:
+            logger.info("Configuration recommendations", recommendations=recommendations)
+
+    except ValidationError:
+        # Re-raise validation errors to fail startup
+        raise
+    except Exception as e:
+        logger.error(
+            "Unexpected error during startup validation",
+            error=str(e),
+            note="Application will continue but may have configuration issues",
+        )
+
+    yield
+
+    # Shutdown
+    logger.info("Shutting down Boards API...")
+
+
+def create_app() -> FastAPI:
+    """Create and configure the FastAPI application."""
+
+    app = FastAPI(
+        title="Boards API",
+        description="Open-source creative toolkit for AI-generated content",
+        version="0.1.0",
+        lifespan=lifespan,
+        debug=settings.debug,
+    )
+
+    # Add tenant routing middleware first (runs before logging)
+    app.add_middleware(TenantRoutingMiddleware)
+
+    # Add logging context middleware
+    app.add_middleware(LoggingContextMiddleware)
+
+    # CORS configuration
+    app.add_middleware(
+        CORSMiddleware,
+        allow_origins=settings.cors_origins,
+        allow_credentials=True,
+        allow_methods=["*"],
+        allow_headers=["*"],
+    )
+
+    # Health check endpoint
+    @app.get("/health")
+    async def health_check():  # pyright: ignore [reportUnusedFunction]
+        """Health check endpoint."""
+        return {"status": "healthy", "version": "0.1.0"}
+
+    # GraphQL endpoint (allow disabling for tests)
+    if not os.getenv("BOARDS_DISABLE_GRAPHQL"):
+        try:
+            from ..graphql.schema import create_graphql_router, validate_schema
+
+            # Validate schema at startup to catch circular reference errors early
+            logger.info("Validating GraphQL schema...")
+            validate_schema()
+
+            # Create and mount the GraphQL router
+            graphql_router = create_graphql_router()
+            app.include_router(graphql_router, prefix="")
+            logger.info("GraphQL endpoint initialized successfully", endpoint="/graphql")
+        except Exception as e:  # pragma: no cover
+            logger.error("Failed to initialize GraphQL endpoint", error=str(e))
+            # Re-raise to fail fast - server should not start with broken GraphQL
+            raise
+
+    # REST API endpoints (for SSE, webhooks, etc.)
+    from .endpoints import jobs, setup, sse, storage, tenant_registration, webhooks
+
+    app.include_router(sse.router, prefix="/api/sse", tags=["SSE"])
+    app.include_router(jobs.router, prefix="/api/jobs", tags=["Jobs"])
+    app.include_router(webhooks.router, prefix="/api/webhooks", tags=["Webhooks"])
+    app.include_router(storage.router, prefix="/api/storage", tags=["Storage"])
+    app.include_router(setup.router, prefix="/api/setup", tags=["Setup"])
+    app.include_router(
+        tenant_registration.router, prefix="/api/tenants", tags=["Tenant Registration"]
+    )
+
+    return app
+
+
+# Create the main application instance
+app = create_app()
+
+if __name__ == "__main__":
+    import uvicorn
+
+    uvicorn.run(
+        "boards.api.app:app",
+        host=settings.api_host,
+        port=settings.api_port,
+        reload=settings.api_reload,
+        log_level=settings.log_level.lower(),
+    )