@tdk-landscape/tdk-cli-core 1.3.15 → 1.3.22

package/engine/docs/C4-ARCHITECTURE-overview.md

@@ -0,0 +1,284 @@
+# TDK Landscape Tilt Architecture - C4 Model
+
+## Level 1: System Context
+
+```mermaid
+graph TB
+    Dev[Developer]
+    Tilt[Tilt Orchestrator]
+    Docker[Docker/OrbStack]
+    Git[Git Repository]
+    Services[Microservices<br/>30+ services]
+    Infra[Infrastructure<br/>Postgres, NATS, Traefik]
+    
+    Dev -->|tilt up| Tilt
+    Tilt -->|watches| Git
+    Tilt -->|builds| Docker
+    Tilt -->|orchestrates| Services
+    Tilt -->|provisions| Infra
+    Docker -->|runs| Services
+    Docker -->|runs| Infra
+    Services -->|uses| Infra
+    
+    style Tilt fill:#f9f,stroke:#333,stroke-width:4px
+    style Dev fill:#bbf,stroke:#333,stroke-width:2px
+    style Services fill:#bfb,stroke:#333,stroke-width:2px
+    style Infra fill:#fbb,stroke:#333,stroke-width:2px
+```
+
+**Purpose**: Tilt automates the entire local development lifecycle - from code changes to running containers.
+
+---
+
+## Level 2: Container Diagram - Tilt SDK Architecture
+
+```mermaid
+graph TB
+    subgraph "Developer Interface"
+        CLI[Tiltfile<br/>Main Entrypoint]
+        Focus[Focus Mode<br/>--focus=user]
+    end
+    
+    subgraph "Core Layer"
+        Utils[Utils Module<br/>I/O, validation, helpers]
+        Manifest[Manifest Module<br/>JSON loading & validation]
+        Registry[Service Registry<br/>APP_RESOURCES definition]
+        DepSync[Dependency Sync<br/>package.json → configs]
+    end
+    
+    subgraph "Provider Layer - Config Generators"
+        Docker[Docker Provider<br/>Dockerfile generation]
+        TS[TypeScript Provider<br/>tsconfig.json]
+        Vite[Vite Provider<br/>vite.config.ts]
+        NPM[Package Config<br/>.npmrc, bunfig.toml]
+    end
+    
+    subgraph "Lifecycle Layer - Orchestration"
+        Libs[Library Manager<br/>Shared packages]
+        DB[Database Provider<br/>Per-service DBs]
+        Orch[Orchestrator<br/>Service coordination]
+        Watch[Watchers<br/>File change detection]
+    end
+    
+    subgraph "Infrastructure"
+        Golden[Golden Image Builder<br/>Base Docker images]
+        Network[Network Manager<br/>Docker network repair]
+        Infisical[Secrets Validator<br/>Infisical integration]
+    end
+    
+    CLI --> Focus
+    CLI --> Utils
+    Focus --> Registry
+    Utils --> Manifest
+    Manifest --> Registry
+    Registry --> DepSync
+    
+    DepSync --> Docker
+    DepSync --> TS
+    DepSync --> Vite
+    DepSync --> NPM
+    
+    Docker --> Libs
+    TS --> Libs
+    Vite --> Libs
+    NPM --> Libs
+    
+    Libs --> DB
+    DB --> Orch
+    Orch --> Watch
+    
+    Orch --> Golden
+    Orch --> Network
+    Orch --> Infisical
+    
+    style CLI fill:#f9f,stroke:#333,stroke-width:4px
+    style Registry fill:#ff9,stroke:#333,stroke-width:3px
+    style Docker fill:#9cf,stroke:#333,stroke-width:3px
+    style Orch fill:#9f9,stroke:#333,stroke-width:3px
+```
+
+---
+
+## Level 3: Component Diagram - Docker Build Pipeline
+
+```mermaid
+graph LR
+    subgraph "Input Detection"
+        Manifest[Service Manifest<br/>JSON file]
+        PkgJSON[package.json<br/>Dependencies]
+        Code[Source Code<br/>src/]
+    end
+    
+    subgraph "Golden Image System"
+        L1Base[L1: OS Base<br/>Alpine + tools]
+        L2Deps[L2: Dependencies<br/>Bun + packages]
+        L4Runtime[L4: Runtime<br/>Backend/Frontend/Migrator]
+    end
+    
+    subgraph "Service-Specific Layers"
+        L3Build[L3: Build Layer<br/>TypeScript compilation]
+        L4Service[L4: Service Runtime<br/>Final container]
+    end
+    
+    subgraph "Output Artifacts"
+        DockerfileApp[Dockerfile.app.autogenerated]
+        DockerfileMigrator[Dockerfile.migrator.autogenerated]
+        Compose[docker-compose.autogenerated.yml]
+    end
+    
+    subgraph "Orchestration"
+        TiltResource[Tilt Resource<br/>service-backend]
+        DockerBuild[Docker Build<br/>Multi-stage]
+        Container[Running Container<br/>Port 4000]
+    end
+    
+    Manifest --> L1Base
+    PkgJSON --> L2Deps
+    Code --> L3Build
+    
+    L1Base --> L2Deps
+    L2Deps --> L3Build
+    L3Build --> L4Service
+    L4Runtime --> L4Service
+    
+    L4Service --> DockerfileApp
+    L4Service --> DockerfileMigrator
+    DockerfileApp --> Compose
+    DockerfileMigrator --> Compose
+    
+    Compose --> TiltResource
+    TiltResource --> DockerBuild
+    DockerBuild --> Container
+    
+    style L1Base fill:#faa,stroke:#333,stroke-width:2px
+    style L2Deps fill:#fca,stroke:#333,stroke-width:2px
+    style L3Build fill:#fea,stroke:#333,stroke-width:2px
+    style L4Service fill:#afa,stroke:#333,stroke-width:2px
+    style Container fill:#9f9,stroke:#333,stroke-width:3px
+```
+
+---
+
+## Level 4A: Code Flow - Manifest-Driven Resource Creation
+
+```mermaid
+sequenceDiagram
+    participant Tilt as Tilt Orchestrator
+    participant Manifest as service.json
+    participant ConfigGen as {service}-config-gen (Tilt Resource)
+    participant Generators as 10+ Config Generators
+    participant Docker as Docker Build
+    
+    Tilt->>Manifest: 1. Scan & Auto-discover
+    Manifest->>ConfigGen: 2. Create per-manifest resource
+    Note over ConfigGen: Resource name: user-backend-config-gen<br/>Label: config-gen, user<br/>Deps: manifest.json, package.json
+    
+    ConfigGen->>Generators: 3. Trigger all generators for THIS manifest
+    Note over Generators: tsconfig.json (Docker)<br/>tsconfig.json (local)<br/>vite.config.ts<br/>Dockerfile.app.autogenerated<br/>.dockerignore<br/>.env.autogenerated<br/>.npmrc (Docker)<br/>.npmrc (local)<br/>bunfig.toml (Docker)<br/>bunfig.toml (local)<br/>nginx.conf (if frontend)<br/>prisma.config.ts (if backend)
+    
+    Generators-->>ConfigGen: 4. Configs generated
+    ConfigGen->>Docker: 5. Trigger Docker build (depends on config-gen)
+    Docker-->>Tilt: 6. Service running
+    
+    Note over Manifest,Docker: ✅ Manifest-driven: 1 manifest → 1 resource → 10 generators<br/>❌ NOT bulk: 100 manifests → 1 loop → 1000 configs
+```
+
+### Architecture Evolution
+
+**Before (Bulk Generation)** - Lines 112-251 of apply.star:
+```starlark
+for resource in ALL_RESOURCES:  # 30 services
+    generate_vite(resource)      # Loop 1
+for resource in ALL_RESOURCES:
+    generate_dockerfile(resource) # Loop 2
+for resource in ALL_RESOURCES:
+    generate_tsconfig(resource)   # Loop 3
+# ... 7 more loops
+
+# ❌ Problems:
+# - No per-resource tracking in Tilt UI
+# - Change 1 manifest → regenerate ALL configs
+# - Can't see which config belongs to which service
+```
+
+**After (Manifest-Driven)** - Lines 114-136 of apply.star:
+```starlark
+for resource in resource_resources:
+    manifest = load_manifest(resource)
+    
+    # Create ONE Tilt resource that generates ALL configs for THIS service
+    config_gen_resource = ManifestResource.create_config_resource(
+        resource_name, resource, manifest, backend_manifest, ctx
+    )
+    # Result: user-management-backend-config-gen (visible in Tilt UI)
+
+# ✅ Benefits:
+# - 30 granular Tilt resources (one per service)
+# - Change 1 manifest → regenerate 10 configs for THAT service only  
+# - Clear dependency: Docker build depends on config-gen resource
+```
+
+### Tilt UI Visibility
+
+In Tilt dashboard you now see per-manifest resources:
+
+```
+📁 config-gen (label)
+  ├─ user-management-backend-config-gen ✅
+  ├─ user-management-frontend-config-gen ✅
+  ├─ identity-management-backend-config-gen ✅
+  ├─ identity-management-frontend-config-gen ✅
+  └─ staff-management-backend-config-gen ✅
+
+📁 user (label)
+  ├─ user-management-backend-config-gen ⬆️ (dependency)
+  ├─ user-management-backend ✅ (depends on config-gen)
+  ├─ user-management-frontend-config-gen ⬆️ (dependency)
+  └─ user-management-frontend ✅ (depends on config-gen)
+```
+
+### Dependency Chain
+
+```
+manifest.json  ─┐
+package.json   ─┼─→ {service}-config-gen ───→ {service} (Docker)
+                │    (10+ generators)           (depends on configs)
+                │
+Infrastructure ─┘
+  golden-image (L1-L4 build cache, not a runtime dep) │ postgres (Free) │
+  verdaccio (Premium, requires TDK_LICENSE_KEY) - each independent,
+  gated by should_enable(), see infra-loader.star
+```
+
+---
+
+## Level 4B: Code Flow - Dependency Synchronization
+
+```mermaid
+sequenceDiagram
+    participant Dev as Developer
+    participant File as package.json
+    participant Tilt as Tilt Watcher
+    participant DepSync as Dependency Sync
+    participant TS as TSConfig Generator
+    participant Vite as Vite Generator
+    participant Docker as Docker Generator
+    participant Service as Service Container
+    
+    Dev->>File: bun add @tdk-landscape/eventing
+    File->>Tilt: File change detected
+    Tilt->>DepSync: Trigger sync
+    DepSync->>DepSync: Extract internal deps
+    DepSync->>DepSync: Resolve library paths
+    DepSync->>TS: Generate tsconfig.json paths
+    DepSync->>Vite: Generate vite.config.ts aliases
+    DepSync->>Docker: Update COPY statements
+    Docker->>Docker: Regenerate Dockerfile
+    Docker->>Service: Rebuild container
+    Service-->>Dev: Service restarted with new dependency
+    
+    Note over DepSync,Docker: Convention over Configuration:<br/>@tdk-landscape/platform-x → platform/packages/platform-x/src
+```
+
+---
+

package/engine/docs/C4-ARCHITECTURE.md

@@ -0,0 +1,17 @@
+# TDK Landscape Tilt Architecture - C4 Model
+
+This document was split to keep each file focused and maintainable.
+
+## Architecture Docs Index
+
+- [C4 Overview (Context, Containers, Components, Core Flows)](./C4-ARCHITECTURE-overview.md)
+- [C4 Operations (Detailed Breakdown, Principles, Performance, Boundaries)](./C4-ARCHITECTURE-operations.md)
+
+## Why split
+
+- The original file had grown to 678 lines and mixed conceptual and operational concerns.
+- Splitting improves navigation and reduces merge conflicts when updating documentation.
+
+## Backward compatibility
+
+- Existing links to `C4-ARCHITECTURE.md` still work and now point to this index.

package/engine/docs/GOLDEN_IMAGE_INTEGRATION.md

@@ -0,0 +1,349 @@
+# Golden Image Integration Guide
+
+## Overview
+
+The TDK Landscape monorepo now includes a **Golden Image** workflow that significantly reduces service build times from 60+ seconds to **<5 seconds** by pre-building heavy dependencies into a base Docker image.
+
+## Architecture Integration
+
+### 1. Provider Layer (`.tilt/providers/docker/`)
+
+**File**: `.tilt/providers/docker/golden-image.star`
+
+The golden image provider follows the established Tilt SDK pattern:
+
+```starlark
+GoldenImage = struct(
+    build = _build_golden_image,              # Builds the base image
+    check_exists = _check_golden_image_exists, # Validates existence
+    get_reference = _get_golden_image_reference, # Returns image name
+    generate_dockerfile = _generate_golden_dockerfile # Generates Dockerfile
+)
+```
+
+**Exported via**: `.tilt/providers/docker/index.star`
+```starlark
+Docker = struct(
+    # ... existing functions
+    golden_image = GoldenImage,
+)
+```
+
+### 2. Infrastructure Layer (`.tilt/provisioner/infra-loader.star`)
+
+The golden image is treated as **infrastructure**, building before application services:
+
+```starlark
+def load_all_infrastructure(should_enable, fix_docker_networks_fn, docker_provider, write_file_fn):
+    # Initialize networks first
+    _init_networks(fix_docker_networks_fn)
+    
+    # Build golden image before other infrastructure
+    _generate_golden_dockerfile(should_enable, docker_provider, write_file_fn)
+    golden_image_resource = _load_golden_image(should_enable, docker_provider)
+    
+    # Load other infrastructure (database, verdaccio, etc.)
+    ...
+    
+    return golden_image_resource
+```
+
+**Key Functions**:
+- `_load_golden_image()` - Creates `local_resource` for building the image
+- `_generate_golden_dockerfile()` - Auto-generates `docker/golden.Dockerfile` if missing
+
+### 3. Orchestrator Integration (`.tilt/provisioner/orchestrator/apply.star`)
+
+Services automatically depend on the golden image:
+
+```starlark
+# Infrastructure dependencies
+infra_deps = []
+if should_enable('database-management'):
+    infra_deps.extend(['postgres', 'provision-db-' + s_name])
+if should_enable('verdaccio'):
+    infra_deps.append('verdaccio')
+if should_enable('golden-image') and ctx.get('golden_image_resource'):
+    infra_deps.append(ctx['golden_image_resource'])  # ← Added
+```
+
+This ensures the golden image builds before any service Dockerfiles are processed.
+
+### 4. Layer Generator Updates (`.tilt/providers/docker/layers/l1_base_layers.star`)
+
+The L1 base layer now intelligently uses the golden image:
+
+```starlark
+GOLDEN_IMAGE = "TDK Landscape-base:latest"
+
+def L1_generate_os_base(base_image = None, maintainer = "TDK Landscape", use_golden = True):
+    if base_image == None:
+        base_image = GOLDEN_IMAGE if use_golden else ALPINE_BASE
+    
+    # If using golden image, skip dependency installation
+    if base_image == GOLDEN_IMAGE or "TDK Landscape-base" in base_image:
+        return (
+            "FROM " + base_image + " AS l1_os_base\n"
+            + "# Dependencies pre-installed in golden image\n"
+            + "WORKDIR /app\n"
+        )
+    
+    # Standard base with full dependency installation
+    ...
+```
+
+### 5. Configuration (`.tilt/core/registry/config.star`)
+
+Golden image is **enabled by default**. Verdaccio is Premium and off by
+default - it requires TDK_LICENSE_KEY (see project-features.ts):
+
+```starlark
+DEFAULTS = {
+    # Infrastructure defaults
+    'database-management': True,
+    'proxy': True,
+    'verdaccio': False,  # Premium - requires TDK_LICENSE_KEY
+    'golden-image': True,  # ← Added
+    ...
+}
+```
+
+### 6. Tiltfile Integration
+
+The main Tiltfile passes required context:
+
+```starlark
+# Infrastructure loading with golden image support
+GOLDEN_IMAGE_RESOURCE = Infra.load_all(
+    should_enable, 
+    Utils.fix_docker_networks,
+    Docker,                      # ← Docker provider
+    Utils.write_file_if_changed  # ← File writer
+)
+
+# Store in context for orchestrator
+if GOLDEN_IMAGE_RESOURCE:
+    TILT_CONTEXT['golden_image_resource'] = GOLDEN_IMAGE_RESOURCE
+```
+
+## Resource Dependencies Graph
+
+```
+init-networks
+    ↓
+golden-image-build (local_resource)
+    ↓
+database, verdaccio, ... (infrastructure)
+    ↓
+service-backend-1, service-backend-2, ... (applications)
+```
+
+## How It Works
+
+### Build Process
+
+1. **Tilt Startup**:
+   - `Infra.load_all()` is called
+   - `_generate_golden_dockerfile()` creates `docker/golden.Dockerfile` if missing
+   - `_load_golden_image()` registers a `local_resource` named `golden-image-build`
+
+2. **Golden Image Build**:
+   - Resource: `golden-image-build`
+   - Labels: `['infra.docker', 'golden-image']`
+   - Trigger: Manual or when `docker/golden.Dockerfile` changes
+   - Command: `docker build -f docker/golden.Dockerfile -t TDK Landscape-base:latest .`
+
+3. **Service Builds**:
+   - Services declare `resource_deps=['golden-image-build']` (added automatically via `infra_deps`)
+   - Service Dockerfiles generate with `FROM TDK Landscape-base:latest AS l1_os_base`
+   - Only service-specific layers rebuild (dependencies already in base)
+
+### Performance Impact
+
+**Before (without golden image)**:
+- Each service: ~60-90 seconds
+- Total for 5 services: ~5-7 minutes
+- Includes: Alpine base download, Bun installation, Prisma CLI, system deps
+
+**After (with golden image)**:
+- Golden image: ~45 seconds (one-time)
+- Each service: ~3-5 seconds
+- Total for 5 services: ~1 minute
+- Includes: Only service-specific code and dependencies
+
+**Savings**: ~80% reduction in total build time
+
+## Usage
+
+### Enable/Disable
+
+**Via .env**:
+```bash
+# Not needed - controlled via DEFAULTS
+```
+
+**Via CLI**:
+```bash
+tilt up -- --golden-image=false  # Disable
+```
+
+**Via Focus Mode**:
+```bash
+tilt up -- --focus user          # Golden image auto-enabled (in ALWAYS_ENABLED_INFRA)
+```
+
+### Rebuild Golden Image
+
+**Trigger in Tilt UI**:
+1. Find `golden-image-build` resource in left sidebar
+2. Click resource name
+3. Click "Trigger Update" button
+
+**CLI**:
+```bash
+docker build -f docker/golden.Dockerfile -t TDK Landscape-base:latest .
+```
+
+**Automatic Rebuild**:
+- Triggered when `docker/golden.Dockerfile` changes
+- Tilt watches this file via `deps=[GOLDEN_DOCKERFILE]`
+
+### Customize Golden Image
+
+**Option 1: Edit Generated File** (temporary):
+```bash
+vim docker/golden.Dockerfile
+# Trigger rebuild in Tilt
+```
+
+**Option 2: Update Generator** (permanent):
+```bash
+vim .tilt/providers/docker/golden-image.star
+# Edit _generate_golden_dockerfile() function
+# Delete docker/golden.Dockerfile
+# Restart Tilt to regenerate
+```
+
+## File Structure
+
+```
+tdk/
+├── docker/
+│   ├── golden.Dockerfile          # Auto-generated base image
+│   └── README.md                   # Usage guide
+├── .tilt/
+│   ├── providers/
+│   │   └── docker/
+│   │       ├── golden-image.star  # Provider implementation
+│   │       ├── index.star         # Exports GoldenImage
+│   │       └── layers/
+│   │           └── l1_base_layers.star  # Uses golden image
+│   ├── lifecycle/
+│   │   ├── infra-loader.star      # Builds golden image
+│   │   └── orchestrator/
+│   │       └── apply.star         # Adds resource deps
+│   └── core/
+│       └── registry/
+│           └── config.star        # DEFAULTS configuration
+└── Tiltfile                        # Main integration
+```
+
+## Troubleshooting
+
+### Golden Image Not Building
+
+**Symptoms**: Services fail with "image not found: TDK Landscape-base:latest"
+
+**Diagnosis**:
+```bash
+docker image inspect TDK Landscape-base:latest
+```
+
+**Fix**:
+```bash
+docker build -f docker/golden.Dockerfile -t TDK Landscape-base:latest .
+```
+
+### Services Not Using Golden Image
+
+**Symptoms**: Service builds still take 60+ seconds
+
+**Check Generated Dockerfiles**:
+```bash
+grep "FROM TDK Landscape-base" services/*/*/*/Dockerfile.app.autogenerated
+```
+
+**Expected Output**:
+```
+FROM TDK Landscape-base:latest AS l1_os_base
+```
+
+**If Missing**: L1 layer not using golden image - check `use_golden` parameter
+
+### Golden Image Resource Missing
+
+**Symptoms**: `ctx.get('golden_image_resource')` returns None
+
+**Check**:
+1. Verify `'golden-image': True` in DEFAULTS
+2. Check Tiltfile stores `GOLDEN_IMAGE_RESOURCE` in `TILT_CONTEXT`
+3. Restart Tilt
+
+## Testing
+
+Run integration tests:
+```bash
+./test-golden-image.sh
+```
+
+**Tests**:
+1. ✅ Provider files exist
+2. ✅ DEFAULTS configuration
+3. ✅ Docker provider exports GoldenImage
+4. ✅ Infrastructure loader integrates golden image
+5. ✅ Tiltfile passes Docker provider
+6. ✅ L1 layer supports golden image
+
+## Advanced: Disabling Per-Service
+
+To opt-out a specific service from using the golden image:
+
+**In `golden_docker_generator_v2.star`**:
+```starlark
+def L4_generate_orchestrator(res_path, use_golden_image = True, ...):
+    parts.append(L1_generate_os_base(use_golden=use_golden_image))
+    ...
+```
+
+**In service manifest** (`.tilt/service.manifest.json`):
+```json
+{
+  "useGoldenImage": false
+}
+```
+
+## OrbStack Compatibility
+
+The golden image is optimized for **OrbStack**:
+- No registry push/pull needed
+- Images stored in local OrbStack cache
+- Fast layer sharing between services
+- Resource-efficient (single base image shared)
+
+**Does NOT require**:
+- External Docker registry
+- Image push/pull credentials
+- Network bandwidth for base image
+
+## Next Steps
+
+1. **Extend Golden Image**: Add more common dependencies (e.g., PostgreSQL client, Redis tools)
+2. **Version Golden Image**: Tag with version numbers for reproducibility
+3. **Multi-Architecture**: Support ARM64 and AMD64 variants
+4. **Layer Optimization**: Further reduce golden image size
+
+## References
+
+- [Docker Multi-Stage Builds](https://docs.docker.com/build/building/multi-stage/)
+- [Tilt Local Resources](https://docs.tilt.dev/local_resource.html)
+- [OrbStack Performance](https://orbstack.dev/features)