@flisk-sh/flisk 0.0.0-stage → 1.3.189

package/discovery/AGENTS.md

@@ -0,0 +1,362 @@
+# AGENTS.md - Tilt Discovery System
+
+## Purpose
+
+This folder contains the **service discovery system** for Flisk's Tilt infrastructure. It scans the repository for service manifests and creates Tilt resources.
+
+## Architecture: Two-Layer Scanning with Validation
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                 DISCOVERY SYSTEM                            │
+├─────────────────────────────────────────────────────────────┤
+│                                                             │
+│  LAYER 1: AUTO DISCOVERY (Orchestrator)                      │
+│  ┌─────────────────────────────────────────────────────┐   │
+│  │  discovery_orchestrator.star                        │   │
+│  │  ├── Orchestrates the discovery process             │   │
+│  │  ├── Multi-root scanning (Phase 2)                  │   │
+│  │  ├── Root overlap validation (Phase 2)              │   │
+│  │  ├── Loads and validates manifests (Phase 1.5)      │   │
+│  │  └── Logs validation warnings with file paths       │   │
+│  └─────────────────────────────────────────────────────┘   │
+│                            │                                │
+│                            ▼                                │
+│  LAYER 2: MANUAL SCANNER (Folder-Specific)                 │
+│  ┌─────────────────────────────────────────────────────┐   │
+│  │  json_manifest_scanner.star                         │   │
+│  │  ├── Low-level shell `find` command                 │   │
+│  │  ├── Scans one root per call (reused for Phase 2)   │   │
+│  │  └── Returns list of manifest paths                 │   │
+│  └─────────────────────────────────────────────────────┘   │
+│                            │                                │
+│                            ▼                                │
+│  LAYER 3: VALIDATION (Phase 1.5)                           │
+│  ┌─────────────────────────────────────────────────────┐   │
+│  │  validation.star                                    │   │
+│  │  ├── validate(manifest) → returns issues[]          │   │
+│  │  ├── Validates: appName, appType, domain, port      │   │
+│  │  ├── Validates: features, dependsOn      │   │
+│  │  └── Lenient mode: warns but continues discovery    │   │
+│  └─────────────────────────────────────────────────────┘   │
+│                            │                                │
+│                            ▼                                │
+│  LAYER 4: REGISTRY (Service Registry)                        │
+│  ┌─────────────────────────────────────────────────────┐   │
+│  │  registry.star                                      │   │
+│  │  ├── get_app_resources()                             │   │
+│  │  ├── get_resource_by_name(name)                    │   │
+│  │  └── Exports to autogenerated files               │   │
+│  └─────────────────────────────────────────────────────┘   │
+│                                                             │
+└─────────────────────────────────────────────────────────────┘
+```
+
+## Phase 2: Universal Discovery (Implemented)
+
+**Status:** ✅ **IMPLEMENTED** - Multi-root discovery with validation and explicit domain extraction
+
+### What Changed
+
+Phase 2 extends discovery from single-root (`services/product/`) to multi-root scanning:
+
+**Scan Roots Configuration (`constants.star`):**
+```starlark
+DISCOVERY_SCAN_ROOTS = [
+    RESOURCES_ROOT,                           # services/product/*/*/
+    "../../../../services/platform",         # services/platform/*/*/
+    "../../../../shared-product-engineering",  # shared-product-engineering/*/
+    "../../../../cli-platform-engineering",    # cli-platform-engineering/*/
+]
+```
+
+**Domain Extraction Patterns (`SCAN_ROOT_PATTERNS`):**
+```starlark
+SCAN_ROOT_PATTERNS = {
+    RESOURCES_ROOT: {"domain_depth": -3},  # Domain-grouped (product)
+    "../../../../services/platform": {"domain_depth": -2},  # Flat
+    # ... etc
+}
+```
+
+**Performance Guardrails:**
+- `MAX_SCAN_TIMEOUT_MS = 5000` - Warn if scan takes >5 seconds
+- `MAX_MANIFESTS_PER_ROOT = 50` - Warn if root has >50 manifests
+
+### Architecture Changes
+
+Phase 2 adds two key features to the orchestrator:
+
+1. **Root Overlap Validation** - Prevents duplicate scanning from misconfiguration
+   ```starlark
+   def _validate_scan_roots(roots):
+       # Warns if one root is subdirectory of another
+   ```
+
+2. **Multi-Root Scanning** - Iterates over all configured roots
+   ```starlark
+   for root in DISCOVERY_SCAN_ROOTS:
+       paths = discover_json_manifests(root)
+       # ... deduplicate and process
+   ```
+
+3. **Explicit Domain Extraction** - Uses configured patterns instead of hardcoded assumptions
+   ```starlark
+   pattern = SCAN_ROOT_PATTERNS.get(matching_root, {"domain_depth": -2})
+   domain = path_parts[pattern["domain_depth"]]
+   ```
+
+### Expected Service Counts
+
+| Root | Expected Manifests | Structure |
+|------|---------------------|-----------|
+| `services/product/` | ~26 | Domain-grouped (`{domain}/{service}/`) |
+| `services/platform/` | ~9 | Flat (`{service}/`) |
+| `shared-product-engineering/` | ~3 | Flat (`{service}/`) |
+| `cli-platform-engineering/` | ~1 | Flat (`{service}/`) |
+| **Total** | **~39** | - |
+
+### "Pool of Candidates" Behavior
+
+Phase 2 discovers **all manifests** from all scan roots, creating a "pool of candidates." Not every candidate will be ready to run, and that's expected:
+
+- Some platform services are works-in-progress
+- Some services may lack complete infrastructure (Dockerfile, configs)
+- Tilt will show clear error messages for services with missing infrastructure
+- Developers can see what's available vs what's broken in Tilt UI
+
+**This is intentional** - Starlark has limited filesystem capabilities for pre-flight validation. Services fail naturally at build/orchestration time with clear Tilt error messages.
+
+### Example Phase 2 Discovery Output
+
+```
+📄 FIRST PASS: Loading services from JSON manifests...
+📁 Scanned 26 manifests from ../../../../services/product
+📁 Scanned 9 manifests from ../../../../services/platform
+📁 Scanned 3 manifests from ../../../../shared-product-engineering
+📁 Scanned 1 manifests from ../../../../cli-platform-engineering
+⚠️  Validation: services/product/order/management/manifest.json: appName too short
+✅ Discovery complete: 38/39 manifests valid
+⚠️  1 manifest(s) with 1 validation warning(s)
+```
+
+### Rollback to Phase 1
+
+If issues arise with Phase 2:
+
+```bash
+# Restore Phase 1 behavior
+tilt down
+git checkout discovery/constants.star
+git checkout discovery/discovery_orchestrator.star
+tilt up
+```
+
+This reverts to single-root scanning of `services/product/` only.
+
+### Troubleshooting Phase 2
+
+**Issue: Warning about overlapping roots**
+```
+⚠️  Warning: ../../../../services/product/user is subdirectory of ../../../../services/product
+```
+**Solution:** Check `DISCOVERY_SCAN_ROOTS` in `constants.star` - one root should not be inside another.
+
+**Issue: Large root warning**
+```
+⚠️  Large root: ../../../../services/product has 60 manifests (max: 50)
+```
+**Solution:** Either increase `MAX_MANIFESTS_PER_ROOT` or verify the root path is correct.
+
+**Issue: New services not appearing in Tilt UI**
+**Solution:**
+1. Check manifest exists in new root: `find services/platform -name "*.manifest.json"`
+2. Verify manifest is valid JSON
+3. Check Tilt logs: `tilt logs | grep "Scanned"`
+4. Verify domain extraction: Check Tilt UI grouping
+
+---
+
+## Phase 1.5: Validation Integration (Complete)
+
+**Status:** ✅ Complete - Validation integrated into discovery flow
+
+### Validation Behavior
+
+The discovery system now validates all manifests during the scan process:
+
+1. **When:** After `ManifestLoader.load_from_file()` successfully loads a manifest
+2. **What:** Validates required fields (appName, appType, domain, port, features, replicas)
+3. **How:** Lenient mode - logs warnings but continues discovery
+4. **Summary:** Discovery completion shows validation stats
+
+### Validation Rules
+
+| Field | Validation |
+|-------|------------|
+| `appName` | Required, 3-64 chars, kebab-case (lowercase, numbers, hyphens) |
+| `appType` | Required, must be: backend, frontend, library, migrator, sdk, worker, bring-your-own |
+| `domain` | Required, must be a valid domain from `constants.star` |
+| `port` | Required, integer 3000-9999, appType-specific ranges enforced |
+| `features` | Optional, each must be a valid feature from `constants.star` |
+| `replicas` | Optional, integer 1-10 |
+
+### Example Discovery Output
+
+```
+📄 FIRST PASS: Loading services from JSON manifests...
+⚠️  Validation: services/product/order/management/manifest.json: appName too short: 'ab'. Minimum 3 characters
+⚠️  Validation: services/product/user/provider/manifest.json: Invalid appType: microservice. Must be one of: backend, frontend...
+✅ Discovery complete: 24/26 manifests valid
+⚠️  2 manifest(s) with 3 validation warning(s)
+```
+
+## Troubleshooting
+
+### Validation Warnings
+
+**Issue:** Manifest shows validation warnings but service still loads
+```
+⚠️  Validation: services/product/.../manifest.json: appName too short: 'x'
+```
+
+**Solution:** Validation runs in lenient mode. Fix the warning to ensure future compatibility:
+- Check manifest follows all validation rules above
+- Run `tilt logs` to see all validation issues for a manifest
+- View the manifest file path in the warning message
+
+### Discovery Not Finding Services
+
+**Issue:** Services not appearing in Tilt UI
+
+**Solution:**
+1. Check manifest file exists: `services/product/{domain}/{service}/manifest.json`
+2. Verify JSON is valid: `cat manifest.json | python -m json.tool`
+3. Check for validation errors in `tilt logs`
+4. Ensure domain is not in `OUT_OF_SCOPE_DOMAINS` (see `spec.master`)
+
+### Validation Errors in Strict Mode
+
+**Note:** Currently validation runs in **lenient mode** (warnings only). If strict mode is needed:
+- Modify `discovery_orchestrator.star` to `continue` on validation issues
+- Or add `STRICT_VALIDATION = True` config option
+
+---
+
+## Key Files
+
+### Layer 1: Auto Discovery Orchestrator
+- **`discovery_orchestrator.star`** - Main orchestrator
+  - `initialize_discovery(cache, second_pass=False)` - Load all services
+  - `_scan_services()` - Scan and normalize all manifests
+  - Calls manual scanner, processes results
+
+### Layer 2: Manual Folder-Specific Scanner
+- **`json_manifest_scanner.star`** - Low-level JSON manifest finder
+  - `discover_json_manifests(root_path)` - Uses shell `find` command
+  - Folder-specific: scans one root at a time
+  - Returns list of manifest file paths
+
+### Layer 3: Service Registry
+- **`registry.star`** - Service registry and export
+  - `get_app_resources()` - Get all discovered services
+  - `get_resource_by_name(name)` - Get specific service
+  - `generate_product_snapshot_topology()` - Export to autogenerated files
+
+### Configuration & Constants
+- **`constants.star`** - Paths and constants
+  - `RESOURCES_ROOT` - "../../../../services/product"
+  - `DISCOVERY_SCAN_ROOTS` - Array of scan roots (Phase 2)
+  - `SCAN_ROOT_PATTERNS` - Domain extraction patterns per root (Phase 2)
+  - `MAX_SCAN_TIMEOUT_MS` - Performance guardrail (Phase 2)
+  - `MAX_MANIFESTS_PER_ROOT` - Performance guardrail (Phase 2)
+  - `PRODUCT_SNAPSHOT_DIR` - Where to write autogenerated files
+
+### Supporting Files
+- **`config.star`** - Global configuration loading
+- **`libraries.star`** - Library autodiscovery (DDD libs, platform libs)
+- **`validation.star`** - Manifest validation rules
+- **`loading.star`** - High-level loading wrapper
+- **`normalization.star`** - Manifest normalization
+
+### Subfolder: manifest/
+**Note:** Discovery-specific manifest logic (not the general `../manifest/`)
+- **`manifest/loading.star`** - Default sync patterns, Prisma checks
+- **`manifest/normalize.star`** - Manifest normalization and discovery
+- **`manifest/validation.star`** - Domain/app type validation
+
+## Common Tasks
+
+### Get all services (Auto Discovery)
+```starlark
+load("./registry.star", "get_app_resources")
+services = get_app_resources()
+```
+
+### Get service by name
+```starlark
+load("./registry.star", "get_resource_by_name")
+svc = get_resource_by_name("order-management-backend")
+```
+
+### Manual scan for manifests (Manual Scanner)
+```starlark
+load("./json_manifest_scanner.star", "discover_json_manifests")
+paths = discover_json_manifests("../../../../services/product")
+# Returns: ["../../../../services/product/.../manifest.json", ...]
+```
+
+### Initialize full discovery (Orchestrator)
+```starlark
+load("./discovery_orchestrator.star", "initialize_discovery")
+_DISCOVERY_CACHE = {"initialized": False}
+initialize_discovery(_DISCOVERY_CACHE)
+```
+
+## Two-Pass Discovery Process
+
+**Pass 1:** Scan JSON manifests and normalize
+1. `discovery_orchestrator.star` calls `json_manifest_scanner.star`
+2. Scanner returns list of manifest paths
+3. Orchestrator loads and normalizes each manifest
+4. Services stored in cache
+
+**Pass 2:** Export to autogenerated Tilt resources
+1. `registry.star` reads from discovery cache
+2. Generates `.tdk/.tdk-out/product_autogenerated_*/` files
+3. Creates Tilt resources from generated files
+
+## Important Notes
+
+1. **Clear Separation:**
+   - **Manual scanner:** Low-level, folder-specific, reusable
+   - **Auto orchestrator:** High-level, coordinates scanning
+   - **Registry:** Consumer of discovery results
+
+2. **Phase 1 Status:** ✅ Complete
+   - Clean separation between scanner and orchestrator
+   - Good naming: `json_manifest_scanner` + `discovery_orchestrator`
+
+3. **Phase 1.5 Status:** ✅ Complete
+   - Validation integrated into discovery flow
+   - Per-service validation with file paths in error messages
+   - Discovery summary shows validation stats
+   - Backward compatible (lenient mode)
+
+4. **Phase 2 Status:** ✅ **IMPLEMENTED**
+   - Multi-root scanning active (`DISCOVERY_SCAN_ROOTS`)
+   - Scans 4 roots: services/product, services/platform, shared-*, cli-*
+   - Root overlap validation prevents misconfiguration
+   - Explicit domain extraction patterns (`SCAN_ROOT_PATTERNS`)
+   - Performance guardrails (timeouts, manifest limits)
+   - "Pool of candidates" approach - all manifests discovered, some may fail at build time
+
+4. **Autogenerated Files:**
+   - `registry.star` writes to `product_autogenerated_resource_manifests/`
+   - These become Tilt resources
+
+## Dependencies
+
+- `../manifest/` - General manifest parsing (constants, loader, parser)
+- `../../platform/docker/` - Docker constants
+- `../../../../spec.master` - Global defaults