@flisk-sh/flisk 0.0.0-stage → 1.3.190

package/engine/topologies/tilt/manifest/README.md

@@ -0,0 +1,679 @@
+# 📋 Manifest Module
+
+**Status:** ✅ All Phases Complete! (Phase 1-5)  
+**Location:** `engine/topologies/tilt/manifest/`  
+**Purpose:** Clean, modular manifest parsing, loading, validation, and discovery integration system
+
+---
+
+## Quick Start
+
+### Basic Usage
+
+```starlark
+# Load the manifest system
+load("../manifest/__init__.star", "Manifest")
+
+# Access constants
+print(Manifest.filename)  # service.json
+print(Manifest.defaults)  # Default field values
+
+# Load and validate a manifest
+result = Manifest.load_from_file("services/product/user/user-management-backend/service.json")
+if result.error:
+    print("Error:", result.error)
+else:
+    manifest = result.manifest
+    
+    # Validate the manifest
+    validation = Manifest.validate(manifest)
+    if validation.valid:
+        print("✅ Valid:", manifest['appName'])
+    else:
+        print("❌ Validation failed:")
+        for error in validation.errors:
+            print("  -", Manifest.format_error(error))
+```
+
+### Validation Example
+
+```starlark
+# Validate a manifest with full reporting
+report = Manifest.get_validation_report(my_manifest)
+print(report)
+
+# Quick check
+if Manifest.is_valid(my_manifest):
+    print("Manifest is valid!")
+
+# Validate against all resources (checks port conflicts, dependencies)
+all_manifests = Manifest.load_all("services/product").manifests
+result = Manifest.validate_dependencies(my_manifest, all_manifests)
+if not result.valid:
+    print("Cross-service validation failed")
+```
+
+---
+
+## Module Structure
+
+```
+manifest/
+├── __init__.star              # Main API facade
+├── constants.star             # Constants and defaults
+├── schema.star                # Field schema definitions
+├── errors.star                # Error handling
+├── loader.star                # File loading and I/O
+├── parser.star                # JSON parsing and normalization
+├── validator.star             # Comprehensive validation
+├── integration.star           # Discovery system integration ⭐ NEW (Phase 4)
+├── PHASE4_DISCOVERY_PATCH.md  # Migration guide ⭐ NEW (Phase 4)
+├── tests/
+│   ├── test_manifest.star     # Unit tests (Phase 1)
+│   ├── test_loader_parser.star  # Integration tests (Phase 2)
+│   ├── test_validator.star    # Validator tests (Phase 3)
+│   └── test_integration.star  # Integration tests (Phase 4) ⭐ NEW
+└── README.md                  # This file
+```
+
+---
+
+## API Reference
+
+### Constants
+
+| Export | Description |
+|--------|-------------|
+| `Manifest.filename` | Manifest filename constant |
+| `Manifest.defaults` | Default field values |
+| `Manifest.app_types` | Valid app types |
+| `Manifest.stacks` | Valid stacks |
+| `Manifest.features` | Valid features |
+| `Manifest.port_ranges` | Port ranges by app type |
+| `Manifest.autogenerated_paths` | Autogenerated config paths |
+
+### Loading Functions ⭐ NEW (Phase 2)
+
+| Function | Description | Example |
+|----------|-------------|---------|
+| `load_from_file(path)` | Load manifest from file | `Manifest.load_from_file("path/to/manifest.json")` |
+| `load_from_path(resource_path)` | Load from resource directory | `Manifest.load_from_path("services/product/user/my-resource")` |
+| `load_all(root, filters)` | Load all with optional filters | `Manifest.load_all("services/product", {'appType': 'backend'})` |
+| `get_manifest_path(path)` | Get expected manifest path | `Manifest.get_manifest_path("services/product/user/my-resource")` |
+| `clear_cache()` | Clear manifest cache | `Manifest.clear_cache()` |
+| `get_cache_stats()` | Get cache statistics | `Manifest.get_cache_stats()` |
+| `is_cached(path)` | Check if path is cached | `Manifest.is_cached("path/to/manifest.json")` |
+
+**Loading Result Structure:**
+
+```starlark
+result = Manifest.load_from_file("path/to/manifest.json")
+
+# Result struct contains:
+#   result.manifest       - Parsed manifest dict or None
+#   result.error          - Error message or None
+#   result.warnings       - List of warning messages
+#   result.metadata       - Dict with path, from_cache, load_time_ms
+```
+
+**Load All Result Structure:**
+
+```starlark
+result = Manifest.load_all("services/product")
+
+# Result struct contains:
+#   result.manifests      - List of loaded manifests
+#   result.errors         - List of error structs
+#   result.warnings       - List of warning messages
+#   result.stats          - Dict with total_found, successful, failed, filtered_out, from_cache
+```
+
+### Parsing Functions
+
+| Function | Description | Example |
+|----------|-------------|---------|
+| `parse(content, path)` | Parse JSON content | `Manifest.parse(json_string, "path/for/context")` |
+| `normalize(manifest, path)` | Apply defaults and compute fields | `Manifest.normalize(manifest, resource_path)` |
+| `get_normalized(content, path)` | Parse + normalize in one call | `Manifest.get_normalized(json_string, resource_path)` |
+| `extract_resource_path(manifest_path)` | Extract resource path | `Manifest.extract_resource_path("path/to/manifest.json")` |
+| `extract_stack(resource_path)` | Extract stack from path | `Manifest.extract_stack("services/product/user/...")` |
+| `parse_traefik(manifest)` | Parse Traefik config | `Manifest.parse_traefik(manifest)` |
+| `merge_manifests(base, overlay)` | Merge two manifests | `Manifest.merge_manifests(base, overlay)` |
+| `get_description(manifest)` | Get human-readable description | `Manifest.get_description(manifest)` |
+
+**Parse Result Structure:**
+
+```starlark
+result = Manifest.parse(json_content)
+
+# Result struct contains:
+#   result.manifest       - Parsed manifest dict or None
+#   result.error          - Error message or None
+#   result.warnings       - List of warnings (e.g., deprecated fields)
+```
+
+### Validation Functions ⭐ NEW (Phase 3)
+
+| Function | Description | Example |
+|----------|-------------|---------|
+| `validate(manifest, context, level)` | Comprehensive validation | `Manifest.validate(manifest, level='all')` |
+| `is_valid(manifest)` | Quick validity check | `if Manifest.is_valid(manifest): ...` |
+| `get_validation_report(manifest)` | Generate detailed report | `print(Manifest.get_validation_report(m))` |
+| `validate_schema(manifest)` | Schema validation only | `Manifest.validate_schema(manifest)` |
+| `validate_values(manifest)` | Value validation only | `Manifest.validate_values(manifest)` |
+| `validate_dependencies(manifest, all)` | Cross-resource validation | `Manifest.validate_dependencies(m, all_manifests)` |
+
+**Validation Levels:**
+
+| Level | Description |
+|-------|-------------|
+| `'schema'` | Structure and field types |
+| `'values'` | Value ranges, patterns, formats |
+| `'cross_field'` | Cross-field dependencies (e.g., frontend needs backend) |
+| `'cross_resource'` | Port conflicts, dependency existence |
+| `'all'` | All validations (default) |
+
+**Validation Result Structure:**
+
+```starlark
+result = Manifest.validate(manifest, level='all')
+
+# Result struct contains:
+#   result.valid          - True if no errors
+#   result.errors         - List of error structs
+#   result.warnings       - List of warning structs
+#   result.stats          - Dict with fields_checked, rules_passed, rules_failed
+```
+
+**Validation Context:**
+
+```starlark
+context = {
+    'all_manifests': all_manifests,  # For cross-resource checks
+    'resource_path': path,           # For error context
+    'strict': True,                 # Treat warnings as errors
+}
+
+result = Manifest.validate(manifest, context, level='all')
+```
+
+### Schema Functions
+
+| Function | Description | Example |
+|----------|-------------|---------|
+| `get_field(name)` | Get field schema | `Manifest.get_field('port')` |
+| `get_required_fields()` | List required fields | `Manifest.get_required_fields()` |
+| `get_default(name)` | Get default value | `Manifest.get_default('replicas')` |
+| `get_valid_values(name)` | Get enum values | `Manifest.get_valid_values('appType')` |
+| `is_required(name)` | Check if required | `Manifest.is_required('appName')` |
+| `is_deprecated(name)` | Check if deprecated | `Manifest.is_deprecated('apiBasePath')` |
+| `get_all_fields()` | Get all field names | `Manifest.get_all_fields()` |
+
+### Error Handling
+
+| Function | Description |
+|----------|-------------|
+| `format_error(error)` | Format error for display |
+| `group_errors(errors)` | Group errors by category |
+| `has_critical_errors(errors)` | Check for critical errors |
+| `get_error_summary(errors)` | Get error statistics |
+
+### Integration Functions ⭐ NEW (Phase 4)
+
+| Function | Description | Example |
+|----------|-------------|---------|
+| `load_and_validate_integration(path, all, use_new)` | Load with validation | `Manifest.load_and_validate_integration(path, all_manifests)` |
+| `load_all_integration(root, filters, validate)` | Load all with integration | `Manifest.load_all_integration("services/product")` |
+| `convert_to_resource(manifest)` | Convert to discovery format | `Manifest.convert_to_resource(manifest)` |
+| `get_validation_summary(manifests)` | Get validation stats | `Manifest.get_validation_summary(all_manifests)` |
+| `check_port_conflicts(manifests)` | Find port conflicts | `Manifest.check_port_conflicts(manifests)` |
+| `check_dependencies_valid(manifests)` | Validate dependencies | `Manifest.check_dependencies_valid(manifests)` |
+
+**Integration Result Structure:**
+
+```starlark
+result = Manifest.load_all_integration("services/product")
+
+# Result struct contains:
+#   result.manifests      - List of normalized manifests
+#   result.errors         - List of error strings
+#   result.warnings       - List of warning strings
+#   result.stats          - Loading statistics
+#   result.by_name        - Dict mapping appName to manifest
+#   result.by_port        - Dict mapping port to manifest
+```
+
+### Convenience Functions
+
+| Function | Description |
+|----------|-------------|
+| `is_valid_manifest(manifest)` | Quick validation check |
+| `get_resource_type(manifest)` | Get human-readable type description |
+
+---
+
+## Usage Examples
+
+### Example 1: Load and Validate a Single Manifest
+
+```starlark
+load("../manifest/__init__.star", "Manifest")
+
+manifest_path = "services/product/user/user-management-backend/service.json"
+
+result = Manifest.load_from_file(manifest_path)
+
+if result.error:
+    print("❌ Failed to load:", result.error)
+else:
+    manifest = result.manifest
+    
+    # Check if valid
+    if Manifest.is_valid_manifest(manifest):
+        print("✅ Valid manifest:", manifest['appName'])
+        print("   Type:", manifest['appType'])
+        print("   Stack:", manifest['stack'])
+        print("   Port:", manifest['port'])
+    else:
+        print("⚠️  Invalid manifest structure")
+```
+
+### Example 2: Parse and Normalize JSON
+
+```starlark
+json_content = '''
+{
+    "appName": "my-new-service",
+    "appType": "backend",
+    "stack": "user",
+    "port": 4001,
+    "features": ["nats", "prisma"]
+}
+'''
+
+result = Manifest.get_normalized(json_content, "services/product/user/my-new-service")
+
+if result.error:
+    print("Error:", result.error)
+else:
+    manifest = result.manifest
+    
+    # Defaults applied
+    print("Replicas:", manifest['replicas'])  # 1 (default)
+    print("Runtime:", manifest['runtime'])   # bun (default)
+    print("Syncs:", manifest['syncs'])       # auto-computed
+    
+    # Traefik config created
+    print("Traefik pathPrefix:", manifest['traefik']['pathPrefix'])
+```
+
+### Example 3: Load All Backend Services
+
+```starlark
+# Load only backend services
+result = Manifest.load_all(
+    "services/product",
+    filters={
+        'appType': 'backend',
+        'stack': ['user', 'order'],
+    }
+)
+
+print("Found {} manifests ({} successful, {} filtered out)".format(
+    result.stats['total_found'],
+    result.stats['successful'],
+    result.stats['filtered_out']
+))
+
+for manifest in result.manifests:
+    desc = Manifest.get_description(manifest)
+    print("  -", desc)
+```
+
+### Example 4: Handle Loading Errors
+
+```starlark
+result = Manifest.load_all("services/product")
+
+if result.errors:
+    print("⚠️  Some manifests failed to load:")
+    for error in result.errors:
+        formatted = Manifest.format_error(error)
+        print("   -", formatted)
+
+# Check for critical errors
+if Manifest.has_critical_errors(result.errors):
+    print("❌ Critical errors detected - aborting")
+else:
+    print("✅ No critical errors")
+```
+
+### Example 5: Validate a Manifest ⭐ NEW (Phase 3)
+
+```starlark
+manifest = {
+    'appName': 'my-service',
+    'appType': 'backend',
+    'stack': 'user',
+    'port': 4001,
+    'features': ['nats', 'prisma'],
+    'databaseName': 'my_db',
+}
+
+# Full validation
+result = Manifest.validate(manifest, level='all')
+
+if result.valid:
+    print("✅ Manifest is valid!")
+    print("   Fields checked:", result.stats['fields_checked'])
+else:
+    print("❌ Validation failed:")
+    for error in result.errors:
+        print("   -", Manifest.format_error(error))
+```
+
+### Example 6: Cross-Service Validation ⭐ NEW (Phase 3)
+
+```starlark
+# Load all manifests for cross-reference
+all_manifests = Manifest.load_all("services/product").manifests
+
+# Validate a new manifest against existing ones
+new_manifest = {
+    'appName': 'new-frontend',
+    'appType': 'frontend',
+    'stack': 'user',
+    'port': 3005,
+    'backendName': 'user-management-backend',  # References existing service
+}
+
+result = Manifest.validate_dependencies(new_manifest, all_manifests)
+
+if result.valid:
+    print("✅ Dependencies valid!")
+else:
+    print("❌ Dependency issues:")
+    for error in result.errors:
+        print("   -", error.message)
+```
+
+### Example 7: Generate Validation Report ⭐ NEW (Phase 3)
+
+```starlark
+manifest = {
+    'appName': 'test-service',
+    'appType': 'backend',
+    'stack': 'user',
+    'port': 4001,
+}
+
+# Generate detailed report
+report = Manifest.get_validation_report(manifest)
+print(report)
+
+# Output:
+# ======================================================================
+# 📋 MANIFEST VALIDATION REPORT
+# ======================================================================
+# Status: ✅ PASSED
+# 
+# Fields Checked: 15
+# Rules Passed: 14
+# Rules Failed: 0
+# 
+# ======================================================================
+```
+
+### Example 8: Schema-Only Validation ⭐ NEW (Phase 3)
+
+```starlark
+# Just check structure and types
+result = Manifest.validate_schema(manifest)
+
+# Just check values (ranges, patterns)
+result = Manifest.validate_values(manifest)
+
+# Quick check
+if Manifest.is_valid(manifest):
+    print("Manifest is valid!")
+```
+
+### Example 9: Batch Validation with Integration ⭐ NEW (Phase 4)
+
+```starlark
+# Load all manifests with integration
+result = Manifest.load_all_integration("services/product")
+
+print("Loaded {} manifests".format(len(result.manifests)))
+print("Errors: {}".format(len(result.errors)))
+print("Warnings: {}".format(len(result.warnings)))
+
+# Get validation summary
+summary = Manifest.get_validation_summary(result.manifests)
+print("Valid: {}/{}".format(summary['valid'], summary['total']))
+print("Errors by severity: {}".format(summary['by_severity']))
+
+# Check for port conflicts
+conflicts = Manifest.check_port_conflicts(result.manifests)
+if conflicts:
+    print("⚠️  Port conflicts detected:")
+    for conflict in conflicts:
+        print("   -", Manifest.format_error(conflict))
+
+# Check dependencies
+dep_issues = Manifest.check_dependencies_valid(result.manifests)
+if dep_issues:
+    print("⚠️  Dependency issues:")
+    for issue in dep_issues:
+        print("   -", Manifest.format_error(issue))
+```
+
+### Example 10: Discovery Integration ⭐ NEW (Phase 4)
+
+```starlark
+# Use integration layer for discovery system
+result = Manifest.load_and_validate_integration(
+    manifest_path="services/product/user/user-management-backend/service.json",
+    all_manifests=all_manifests,  # For cross-validation
+)
+
+if result.validation:
+    print("Validation: {} errors, {} warnings".format(
+        len([e for e in result.validation.errors if e.severity == 'error']),
+        len(result.validation.warnings)
+    ))
+
+# Convert to discovery resource format
+resource = Manifest.convert_to_resource(result.manifest)
+print("Resource name:", resource['name'])
+print("Resource type:", 'frontend' if resource.get('frontend') else 'backend')
+```
+
+---
+
+## Backward Compatibility
+
+**Old imports still work (deprecated):**
+
+```starlark
+# DEPRECATED - will be removed in Phase 5
+load("../discovery/manifest/constants.star", "MANIFEST_DEFAULTS")
+
+# NEW - recommended
+load("../manifest/__init__.star", "Manifest")
+# Use Manifest.defaults instead of MANIFEST_DEFAULTS
+```
+
+---
+
+## Testing
+
+### Run Unit Tests
+
+```starlark
+# Phase 1 tests (constants, schema, errors)
+load("./manifest/tests/test_manifest.star", "TestManifest")
+TestManifest.run()
+
+# Phase 2 tests (loader, parser)
+load("./manifest/tests/test_loader_parser.star", "TestLoaderParser")
+TestLoaderParser.run()
+
+# Phase 3 tests (validator)
+load("./manifest/tests/test_validator.star", "TestValidator")
+TestValidator.run()
+
+# Phase 4 tests (integration)
+load("./manifest/tests/test_integration.star", "TestIntegration")
+TestIntegration.run()
+```
+
+### Run All Tests
+
+```bash
+# When Tilt test runner is available:
+tilt test manifest/
+
+# Or manually in Tiltfile:
+load("./manifest/tests/test_manifest.star", "TestManifest")
+load("./manifest/tests/test_loader_parser.star", "TestLoaderParser")
+load("./manifest/tests/test_validator.star", "TestValidator")
+load("./manifest/tests/test_integration.star", "TestIntegration")
+
+print("\n=== Phase 1 Tests ===")
+TestManifest.run()
+
+print("\n=== Phase 2 Tests ===")
+TestLoaderParser.run()
+
+print("\n=== Phase 3 Tests ===")
+TestValidator.run()
+
+print("\n=== Phase 4 Tests ===")
+TestIntegration.run()
+```
+
+---
+
+## Migration Phases
+
+| Phase | Status | Description |
+|-------|--------|-------------|
+| Phase 1 | ✅ Complete | Foundation: constants, errors, schema |
+| Phase 2 | ✅ Complete | Core: loader, parser |
+| Phase 3 | ✅ Complete | Validation: validator, rules |
+| Phase 4 | ✅ Complete | Integration: discovery system integration |
+| Phase 5 | ✅ Complete | Cleanup: removed deprecated code |
+
+### Phase 5 Features (✅ Complete)
+
+- **Final Cleanup**
+  - Removed backward compatibility layer
+  - Removed feature flags
+  - Updated discovery.star to use new system exclusively
+  - Deleted migration patch files
+  - All 5 phases complete!
+
+### Phase 4 Features (✅ Complete)
+
+- **Discovery Integration** (`integration.star`)
+  - Integration layer for discovery system
+  - Backward compatibility with legacy discovery
+  - Batch validation across all resources
+  - Port conflict detection across resources
+  - Dependency validation across resources
+  - Validation summary reports
+  - Feature flag for gradual rollout
+
+### Phase 3 Features (✅ Complete)
+
+- **Validation** (`validator.star`)
+  - 4-level validation (schema, values, cross-field, cross-resource)
+  - Schema validation (required fields, types, constraints)
+  - Value validation (port ranges, name formats, patterns)
+  - Cross-field validation (frontend needs backend, prisma needs database)
+  - Cross-resource validation (port conflicts, dependency existence)
+  - Validation reports with detailed statistics
+  - Strict mode (warnings as errors)
+
+### Phase 2 Features (✅ Complete)
+
+- **File Loading** (`loader.star`)
+  - Load single manifest from file
+  - Load from resource directory
+  - Load all with filtering
+  - In-memory caching
+  - Error handling with context
+
+- **Parsing & Normalization** (`parser.star`)
+  - JSON parsing with validation
+  - Automatic field normalization
+  - Default value application
+  - Stack/appType extraction from paths
+  - Traefik configuration generation
+  - Manifest merging for inheritance
+  - Frontend/backend detection
+
+---
+
+## Performance
+
+### Caching
+
+The loader maintains an in-memory cache to avoid redundant file I/O:
+
+```starlark
+# First load reads from disk
+result1 = Manifest.load_from_file("path/to/manifest.json")
+print(result1.metadata['from_cache'])  # False
+
+# Second load uses cache
+result2 = Manifest.load_from_file("path/to/manifest.json")
+print(result2.metadata['from_cache'])  # True
+
+# Clear cache when needed
+Manifest.clear_cache()
+```
+
+### Loading Statistics
+
+```starlark
+result = Manifest.load_all("services/product")
+print("Loaded {} manifests in {} ms".format(
+    result.stats['successful'],
+    result.stats.get('total_time_ms', 0)
+))
+print("From cache:", result.stats['from_cache'])
+```
+
+---
+
+## Future Enhancements
+
+- **Phase 4:** `fetch.star` - Remote manifest fetching
+- **Phase 5:** Cleanup deprecated code
+
+### Completed Phases
+
+- **Phase 1:** Foundation (constants, schema, errors) ✅
+- **Phase 2:** Core (loader, parser) ✅
+- **Phase 3:** Validation (validator, comprehensive rules) ✅
+
+---
+
+## Contributing
+
+When adding new features:
+
+1. Add implementation to appropriate module
+2. Wire up to `__init__.star` facade
+3. Add tests to `tests/` directory
+4. Update this README with examples
+5. Mark Phase X status if applicable
+
+---
+
+## See Also
+
+- [Refactoring Plan](../../../REFACTORING_MANIFEST_SYSTEM.md) - Full refactoring roadmap
+- [AGENTS.md](../../../AGENTS.md) - Agent ecosystem documentation