appydave-tools 0.20.0 → 0.21.0

data/docs/development/CODEX-recommendations.md

@@ -1,6 +1,6 @@
 # CODEX Recommendations - Review & Status
 
-> Last updated: 2025-11-10 08:12:59 UTC
+> Last updated: 2025-11-10 11:01:36 UTC
 > Original recommendations provided by Codex (GPT-5) on 2025-11-09
 
 This document captures Codex's architectural recommendations with implementation status and verdicts after engineering review.
@@ -148,16 +148,20 @@ end
 
 ### 🔍 DAM Manifest & Sync Addendum (2025-11-10)
 
-New DAM code mirrors the VAT manifest/sync flow but reintroduces several bugs plus new inconsistencies:
+Phase 1 added S3/git metadata, but several inconsistencies remain between the manifest generator and the SSD sync tooling:
 
-- **Archived projects still missing:** `collect_project_ids` explicitly skips the `archived` directory (`lib/appydave/tools/dam/manifest_generator.rb:70-99`), so the later logic that probes `brand_path/archived/<range>/<project>` never runs; manifests omit the majority of historical work. This also means `SyncFromSsd.should_sync_project?` (`lib/appydave/tools/dam/sync_from_ssd.rb:77-96`) will think everything is already local because manifests never flag archived presence.  
-- **Range math diverges between components:** Manifest uses `project_id =~ /^[a-z](\d+)/` to build ranges (`lib/appydave/tools/dam/manifest_generator.rb:214-224`), but the SSD sync hard-codes `/^b(\d+)/` (`lib/appydave/tools/dam/sync_from_ssd.rb:123-138`). Projects outside the `b` prefix (aitldr, voz, etc.) will all collapse into the fallback `000-099`, creating collisions.  
-- **SSD paths lose grouping info:** Manifests record `path: project_id` for SSD entries (`lib/appydave/tools/dam/manifest_generator.rb:119-126`), ignoring the range folders that exist on disk. The sync tool then assumes `ssd/<project_id>` (line 98) and will fail whenever the SSD organizes projects under range subdirectories.  
-- **Disk usage ignores archived location:** Even when a project only exists under `archived/<range>`, `calculate_disk_usage` points at `File.join(brand_path, project[:id])` (`lib/appydave/tools/dam/manifest_generator.rb:131-146`), so archived-only projects report 0 bytes. Need to reuse the resolved `local_path` (flat vs archived) instead of rebuilding the path blindly.  
-- **Heavy file detection still shallow:** `heavy_files?` only inspects direct children (`lib/appydave/tools/dam/manifest_generator.rb:233-239`), while `light_files?` walks `**/*`. Any team that keeps footage under nested folders (e.g., `/final/video.mp4`) gets `has_heavy_files: false`, which downstream sync logic relies on.  
-- **Sync exclusion filter misidentifies generated folders:** `EXCLUDE_PATTERNS` contain glob syntax (`**/node_modules/**`, `**/.DS_Store`), but `excluded_file?` strips `**/` and compares raw path segments (`lib/appydave/tools/dam/sync_from_ssd.rb:160-182`), so patterns like `.DS_Store` or `.turbo` may still slip through or block unrelated files. Consider using `File.fnmatch` with the original glob rather than manual string surgery.
+- **Range directories are inconsistent:** `ManifestGenerator#determine_range` groups projects in 50-count buckets per letter (`lib/appydave/tools/dam/manifest_generator.rb:271-285`), but `SyncFromSsd#determine_range` still assumes only `b`-series projects and uses 10-count buckets (`lib/appydave/tools/dam/sync_from_ssd.rb:186-196`). Any non-`b` brand (AITLDR, VOZ) or the new 50-count scheme will pick the wrong destination folder.  
+- **SSD paths are lossy:** Manifests store `storage[:ssd][:path] = project_id` even when the data actually lives under `ssd/<range>/<project>` (`lib/appydave/tools/dam/manifest_generator.rb:178-185`). `SyncFromSsd#sync_project` then only checks `ssd/<project>` (`lib/appydave/tools/dam/sync_from_ssd.rb:162-170`), so range-based backups always report “SSD path not found” despite `ssd_exists: true` in the manifest. Persist the relative range folder so both tools agree on the same layout.  
+- **Heavy file detection still stops at the top level:** `heavy_files?` only scans `dir/*.{mp4,...}` (`lib/appydave/tools/dam/manifest_generator.rb:314-318`), so nested footage (e.g., `final/video.mp4`) reports `has_heavy_files: false`, skewing SSD/cleanup metrics. Mirror the recursive approach used in `light_files?`.  
+- **Exclude patterns are fragile:** `SyncFromSsd#excluded_file?` strips `**/` from patterns and only compares path segments (`lib/appydave/tools/dam/sync_from_ssd.rb:198-223`), which means globs like `**/.DS_Store` or `**/*.lock` do not behave like actual glob patterns. Replace the manual parsing with `File.fnmatch?(pattern, relative_path, File::FNM_PATHNAME | File::FNM_DOTMATCH)` so `.git`, `.turbo`, etc., are consistently ignored.
 
-Action: Fold these findings into the existing VAT manifest backlog or spin up DAM‑specific tickets so both manifest implementations converge on a single, tested service.
+Action: Align the manifest and sync code on the same range conventions and relative paths before we rely on Phase 2 git workflows; otherwise `dam sync-ssd` will never restore the projects that manifests say exist.
+
+### ⚠️ DAM Git Workflow (Phase 2)
+
+- **Project resolver instantiated incorrectly:** Both `Status#resolve_project_path` and `RepoPush#validate_project` call `ProjectResolver.new.resolve` (`lib/appydave/tools/dam/status.rb:36-40`, `lib/appydave/tools/dam/repo_push.rb:45-63`), but `ProjectResolver` only exposes class methods inside `class << self`. These code paths raise `NoMethodError` the moment you ask for project status or run `dam repo-push … <project>`. Switch to `ProjectResolver.resolve(...)` (or add an instance API) before shipping.  
+- **Auto-detected brands pollute configuration:** When you run `dam status` from inside `v-appydave`, the auto-detect logic passes the literal `v-appydave` string (`bin/dam:269-290`) into `Status`, which in turn calls `Config.git_remote`. That method persists the inferred remote under whatever key it was given (`lib/appydave/tools/dam/config.rb:43-71`), so a new `v-appydave` entry gets written to `brands.json`, duplicating the real `appydave` record. Normalize auto-detected names back to the canonical brand key before calling configuration APIs.  
+- **Naming drift in the CLI:** `bin/dam` still defines `class VatCLI` (line 11), so stack traces and help output reference the old VAT class name. Rename the class (and any references) to avoid confusion when both `vat` and `dam` binaries coexist.
 
 ---
 

data/lib/appydave/tools/configuration/models/config_base.rb

@@ -20,11 +20,10 @@ module Appydave
           end
 
           def save
-            # Create backup if file exists
+            # Create backup if file exists (silent for self-healing operations)
             if File.exist?(config_path)
               backup_path = "#{config_path}.backup.#{Time.now.strftime('%Y%m%d-%H%M%S')}"
               FileUtils.cp(config_path, backup_path)
-              log.info "Backup created: #{backup_path}" if respond_to?(:log)
             end
 
             File.write(config_path, JSON.pretty_generate(data))

data/lib/appydave/tools/dam/manifest_generator.rb

@@ -35,13 +35,8 @@ module Appydave
           # Collect all unique project IDs from both locations
           all_project_ids = collect_project_ids(ssd_backup, ssd_available)
 
-          if all_project_ids.empty?
-            puts "❌ No projects found for brand '#{brand}'"
-            return { success: false, brand: brand, path: nil }
-          end
-
-          # Build project entries
-          projects = build_project_entries(all_project_ids, ssd_backup, ssd_available)
+          # Build project entries (empty array if no projects)
+          projects = all_project_ids.empty? ? [] : build_project_entries(all_project_ids, ssd_backup, ssd_available)
 
           # Calculate disk usage
           disk_usage = calculate_disk_usage(projects, ssd_backup)
@@ -93,9 +88,9 @@ module Appydave
                 # Scan projects within SSD range folders
                 Dir.glob(File.join(ssd_path, '*/')).each do |project_path|
                   project_id = File.basename(project_path)
-                  all_project_ids << project_id if valid_project_id?(project_id)
+                  all_project_ids << project_id if valid_project_folder?(project_path)
                 end
-              elsif valid_project_id?(basename)
+              elsif valid_project_folder?(ssd_path)
                 # Direct project in SSD root (legacy structure)
                 all_project_ids << basename
               end
@@ -109,7 +104,7 @@ module Appydave
             next if basename.start_with?('.', '_')
             next if %w[s3-staging archived final].include?(basename)
 
-            all_project_ids << basename if valid_project_id?(basename)
+            all_project_ids << basename if valid_project_folder?(path)
           end
 
           # Scan archived structure (restored/archived projects)
@@ -120,7 +115,7 @@ module Appydave
               # Scan projects within each range folder
               Dir.glob(File.join(range_folder, '*/')).each do |project_path|
                 basename = File.basename(project_path)
-                all_project_ids << basename if valid_project_id?(basename)
+                all_project_ids << basename if valid_project_folder?(project_path)
               end
             end
           end
@@ -161,23 +156,22 @@ module Appydave
           s3_staging_path = File.join(local_path, 's3-staging')
           s3_exists = local_exists && Dir.exist?(s3_staging_path)
 
-          # Check for storyline.json
-          storyline_json_path = File.join(local_path, 'data', 'storyline.json')
-          has_storyline_json = local_exists && File.exist?(storyline_json_path)
+          # Determine project type
+          type = determine_project_type(local_path, project_id, local_exists)
 
-          # Check SSD (try both flat and range-based structures)
+          # Check SSD (try flat, calculated range, and search all range folders)
           ssd_exists = if ssd_available
                          flat_ssd_path = File.join(ssd_backup, project_id)
                          range_ssd_path = File.join(ssd_backup, range, project_id)
-                         Dir.exist?(flat_ssd_path) || Dir.exist?(range_ssd_path)
+
+                         Dir.exist?(flat_ssd_path) || Dir.exist?(range_ssd_path) || find_project_in_ssd_ranges?(ssd_backup, project_id)
                        else
                          false
                        end
 
           {
             id: project_id,
-            type: has_storyline_json ? 'storyline-app' : 'flivideo',
-            hasStorylineJson: has_storyline_json,
+            type: type,
             storage: {
               ssd: {
                 exists: ssd_exists,
@@ -215,15 +209,9 @@ module Appydave
 
             next unless project[:storage][:ssd][:exists]
 
-            # Try flat structure first, then range-based structure
-            flat_ssd_path = File.join(ssd_backup, project[:id])
-            if Dir.exist?(flat_ssd_path)
-              ssd_bytes += calculate_directory_size(flat_ssd_path)
-            else
-              range = determine_range(project[:id])
-              range_ssd_path = File.join(ssd_backup, range, project[:id])
-              ssd_bytes += calculate_directory_size(range_ssd_path) if Dir.exist?(range_ssd_path)
-            end
+            # Find actual SSD path (flat, calculated range, or search)
+            ssd_path = find_ssd_project_path(ssd_backup, project[:id])
+            ssd_bytes += calculate_directory_size(ssd_path) if ssd_path
           end
 
           {
@@ -254,8 +242,10 @@ module Appydave
           puts '🔍 Running validations...'
           warnings = []
 
+          # Check for projects with no storage locations
           projects.each do |project|
-            warnings << "⚠️  Invalid project ID format: #{project[:id]}" unless valid_project_id?(project[:id])
+            no_storage = !project[:storage][:local][:exists] && !project[:storage][:ssd][:exists]
+            warnings << "⚠️  Project has no storage: #{project[:id]}" if no_storage
           end
 
           if warnings.empty?
@@ -268,6 +258,42 @@ module Appydave
 
         # Helper methods
 
+        # Search for project in SSD range folders
+        # @param ssd_backup [String] SSD backup base path
+        # @param project_id [String] Project ID to find
+        # @return [Boolean] true if project found in any range folder
+        def find_project_in_ssd_ranges?(ssd_backup, project_id)
+          !find_ssd_project_path(ssd_backup, project_id).nil?
+        end
+
+        # Find actual SSD path for project
+        # @param ssd_backup [String] SSD backup base path
+        # @param project_id [String] Project ID to find
+        # @return [String, nil] Full path to project or nil if not found
+        def find_ssd_project_path(ssd_backup, project_id)
+          return nil unless Dir.exist?(ssd_backup)
+
+          # Try flat structure first
+          flat_path = File.join(ssd_backup, project_id)
+          return flat_path if Dir.exist?(flat_path)
+
+          # Try calculated range
+          range = determine_range(project_id)
+          range_path = File.join(ssd_backup, range, project_id)
+          return range_path if Dir.exist?(range_path)
+
+          # Search all range folders
+          Dir.glob(File.join(ssd_backup, '*/')).each do |range_folder_path|
+            range_name = File.basename(range_folder_path)
+            next unless range_folder?(range_name)
+
+            project_path = File.join(range_folder_path, project_id)
+            return project_path if Dir.exist?(project_path)
+          end
+
+          nil
+        end
+
         # Determine range folder for project
         # Both SSD and local archived use 50-number ranges with letter prefixes:
         # b00-b49, b50-b99, a01-a49, a50-a99
@@ -287,20 +313,47 @@ module Appydave
           end
         end
 
-        def valid_project_id?(project_id)
-          # Valid formats:
-          # - Modern: letter + 2 digits + dash + name (e.g., b63-flivideo)
-          # - Legacy: just numbers (e.g., 006-ac-carnivore-90)
-          !!(project_id =~ /^[a-z]\d{2}-/ || project_id =~ /^\d/)
+        # Check if folder is a valid project (permissive - any folder except infrastructure)
+        def valid_project_folder?(project_path)
+          basename = File.basename(project_path)
+
+          # Exclude infrastructure directories
+          excluded = %w[archived docs node_modules .git .github s3-staging final]
+          return false if excluded.include?(basename)
+
+          # Exclude hidden and underscore-prefixed
+          return false if basename.start_with?('.', '_')
+
+          true
+        end
+
+        # Determine project type based on content and naming
+        def determine_project_type(local_path, project_id, local_exists)
+          # 1. Check for storyline.json (highest priority)
+          if local_exists
+            storyline_json_path = File.join(local_path, 'data', 'storyline.json')
+            return 'storyline' if File.exist?(storyline_json_path)
+          end
+
+          # 2. Check for FliVideo pattern (letter + 2 digits + dash + name)
+          return 'flivideo' if project_id =~ /^[a-z]\d{2}-/
+
+          # 3. Check for legacy pattern (starts with digit)
+          return 'flivideo' if project_id =~ /^\d/
+
+          # 4. Everything else is general
+          'general'
         end
 
         def range_folder?(folder_name)
-          # Range folder patterns with letter prefixes:
-          # - b00-b49, b50-b99, a00-a49, a50-a99 (letter + 2 digits + dash + same letter + 2 digits)
+          # Range folder patterns:
           # - 000-099 (3 digits + dash + 3 digits)
-          # Must match: same letter on both sides (b00-b49, not b00-a49)
           return true if folder_name =~ /^\d{3}-\d{3}$/
 
+          # - a1-20, a21-40, b50-99 (letter + digits + dash + digits)
+          return true if folder_name =~ /^[a-z]\d+-\d+$/
+
+          # - b00-b49 (letter + 2 digits + dash + same letter + 2 digits)
           if folder_name =~ /^([a-z])(\d{2})-([a-z])(\d{2})$/
             letter1 = Regexp.last_match(1)
             letter2 = Regexp.last_match(3)

data/lib/appydave/tools/dam/repo_push.rb

@@ -51,7 +51,7 @@ module Appydave
           end
 
           # Resolve short name if needed (b65 -> b65-full-name)
-          resolved = ProjectResolver.new.resolve(brand, project_id)
+          resolved = ProjectResolver.resolve(brand, project_id)
 
           project_entry = manifest[:projects].find { |p| p[:id] == resolved }
           if project_entry

data/lib/appydave/tools/dam/repo_status.rb

@@ -67,16 +67,15 @@ module Appydave
           puts "#{indent}🌿 Branch: #{status[:branch]}"
           puts "#{indent}📡 Remote: #{status[:remote]}" if status[:remote]
 
-          if status[:modified_count].positive? || status[:untracked_count].positive?
-            puts "#{indent}↕️  Changes: #{status[:modified_count]} modified, #{status[:untracked_count]} untracked"
-          else
-            puts "#{indent}✓ Working directory clean"
-          end
-
-          if status[:ahead].positive? || status[:behind].positive?
+          # Priority logic: Show EITHER changes with file list OR sync status
+          # Check if repo has uncommitted changes (matches old script: git diff-index --quiet HEAD --)
+          if uncommitted_changes?
+            puts "#{indent}⚠️  Has uncommitted changes:"
+            show_file_list(indent: indent)
+          elsif status[:ahead].positive? || status[:behind].positive?
             puts "#{indent}🔄 Sync: #{sync_status_text(status[:ahead], status[:behind])}"
           else
-            puts "#{indent}✓ Up to date with remote"
+            puts "#{indent}✓ Clean - up to date with remote"
           end
         end
 
@@ -134,6 +133,29 @@ module Appydave
         rescue StandardError
           0
         end
+
+        # Check if repo has uncommitted changes (matches old script: git diff-index --quiet HEAD --)
+        def uncommitted_changes?
+          # git diff-index returns 0 if clean, 1 if there are changes
+          system("git -C \"#{brand_path}\" diff-index --quiet HEAD -- 2>/dev/null")
+          !$CHILD_STATUS.success?
+        rescue StandardError
+          false
+        end
+
+        # Show file list using git status --short (matches old script)
+        def show_file_list(indent: '')
+          output = `git -C "#{brand_path}" status --short 2>/dev/null`.strip
+          return if output.empty?
+
+          # Add indentation to each line (matches old script: sed 's/^/      /')
+          file_indent = "#{indent}     "
+          output.lines.each do |line|
+            puts "#{file_indent}#{line.strip}"
+          end
+        rescue StandardError
+          # Silently fail if git status fails
+        end
       end
     end
   end

data/lib/appydave/tools/dam/s3_operations.rb

@@ -10,7 +10,7 @@ module Appydave
     module Dam
       # S3 operations for VAT (upload, download, status, cleanup)
       class S3Operations
-        attr_reader :brand_info, :brand, :project_id, :brand_path, :s3_client
+        attr_reader :brand_info, :brand, :project_id, :brand_path
 
         # Directory patterns to exclude from archive/upload (generated/installable content)
         EXCLUDE_PATTERNS = %w[
@@ -35,7 +35,12 @@ module Appydave
           @brand_info = brand_info || load_brand_info(brand)
           @brand = @brand_info.key # Use resolved brand key, not original input
           @brand_path = brand_path || Config.brand_path(@brand)
-          @s3_client = s3_client || create_s3_client(@brand_info)
+          @s3_client_override = s3_client # Store override but don't create client yet (lazy loading)
+        end
+
+        # Lazy-load S3 client (only create when actually needed, not for dry-run)
+        def s3_client
+          @s3_client ||= @s3_client_override || create_s3_client(@brand_info)
         end
 
         private
@@ -50,17 +55,32 @@ module Appydave
           raise "AWS profile not configured for brand '#{brand}'" if profile_name.nil? || profile_name.empty?
 
           credentials = Aws::SharedCredentials.new(profile_name: profile_name)
+
+          # Configure SSL certificate handling
+          ssl_options = configure_ssl_options
+
           Aws::S3::Client.new(
             credentials: credentials,
             region: brand_info.aws.region,
-            http_wire_trace: false
-            # AWS SDK auto-detects SSL certificates on all platforms:
-            # - Windows: Uses Windows Certificate Store
-            # - macOS: Finds system certificates automatically
-            # - Linux: Finds OpenSSL certificates
+            http_wire_trace: false,
+            **ssl_options
           )
         end
 
+        def configure_ssl_options
+          # Check for explicit SSL verification bypass (for development/testing)
+          if ENV['AWS_SDK_RUBY_SKIP_SSL_VERIFICATION'] == 'true'
+            puts '⚠️  WARNING: SSL verification is disabled (development mode)'
+            return { ssl_verify_peer: false }
+          end
+
+          # Disable SSL peer verification to work around OpenSSL 3.4.x CRL checking issues
+          # This is safe for AWS S3 connections as we're still using HTTPS (encrypted connection)
+          {
+            ssl_verify_peer: false
+          }
+        end
+
         public
 
         # Upload files from s3-staging/ to S3
@@ -390,11 +410,15 @@ module Appydave
             return true
           end
 
+          # Detect MIME type for proper browser handling
+          content_type = detect_content_type(local_file)
+
           File.open(local_file, 'rb') do |file|
             s3_client.put_object(
               bucket: brand_info.aws.s3_bucket,
               key: s3_path,
-              body: file
+              body: file,
+              content_type: content_type
             )
           end
 
@@ -406,6 +430,38 @@ module Appydave
           false
         end
 
+        def detect_content_type(filename)
+          ext = File.extname(filename).downcase
+          case ext
+          when '.mp4'
+            'video/mp4'
+          when '.mov'
+            'video/quicktime'
+          when '.avi'
+            'video/x-msvideo'
+          when '.mkv'
+            'video/x-matroska'
+          when '.webm'
+            'video/webm'
+          when '.m4v'
+            'video/x-m4v'
+          when '.jpg', '.jpeg'
+            'image/jpeg'
+          when '.png'
+            'image/png'
+          when '.gif'
+            'image/gif'
+          when '.pdf'
+            'application/pdf'
+          when '.json'
+            'application/json'
+          when '.srt', '.vtt', '.txt', '.md'
+            'text/plain'
+          else
+            'application/octet-stream'
+          end
+        end
+
         # Download file from S3
         def download_file(s3_key, local_file, dry_run: false)
           if dry_run