appydave-tools 0.18.1 → 0.18.3

data/docs/development/CODEX-recommendations.md

@@ -1,6 +1,6 @@
 # CODEX Recommendations - Review & Status
 
-> Last updated: 2025-11-10
+> Last updated: 2025-11-10 08:12:59 UTC
 > Original recommendations provided by Codex (GPT-5) on 2025-11-09
 
 This document captures Codex's architectural recommendations with implementation status and verdicts after engineering review.
@@ -146,6 +146,19 @@ end
 
 **Note:** These are legitimate technical debt items, not style preferences. Recommend creating GitHub issues for tracking.
 
+### 🔍 DAM Manifest & Sync Addendum (2025-11-10)
+
+New DAM code mirrors the VAT manifest/sync flow but reintroduces several bugs plus new inconsistencies:
+
+- **Archived projects still missing:** `collect_project_ids` explicitly skips the `archived` directory (`lib/appydave/tools/dam/manifest_generator.rb:70-99`), so the later logic that probes `brand_path/archived/<range>/<project>` never runs; manifests omit the majority of historical work. This also means `SyncFromSsd.should_sync_project?` (`lib/appydave/tools/dam/sync_from_ssd.rb:77-96`) will think everything is already local because manifests never flag archived presence.  
+- **Range math diverges between components:** Manifest uses `project_id =~ /^[a-z](\d+)/` to build ranges (`lib/appydave/tools/dam/manifest_generator.rb:214-224`), but the SSD sync hard-codes `/^b(\d+)/` (`lib/appydave/tools/dam/sync_from_ssd.rb:123-138`). Projects outside the `b` prefix (aitldr, voz, etc.) will all collapse into the fallback `000-099`, creating collisions.  
+- **SSD paths lose grouping info:** Manifests record `path: project_id` for SSD entries (`lib/appydave/tools/dam/manifest_generator.rb:119-126`), ignoring the range folders that exist on disk. The sync tool then assumes `ssd/<project_id>` (line 98) and will fail whenever the SSD organizes projects under range subdirectories.  
+- **Disk usage ignores archived location:** Even when a project only exists under `archived/<range>`, `calculate_disk_usage` points at `File.join(brand_path, project[:id])` (`lib/appydave/tools/dam/manifest_generator.rb:131-146`), so archived-only projects report 0 bytes. Need to reuse the resolved `local_path` (flat vs archived) instead of rebuilding the path blindly.  
+- **Heavy file detection still shallow:** `heavy_files?` only inspects direct children (`lib/appydave/tools/dam/manifest_generator.rb:233-239`), while `light_files?` walks `**/*`. Any team that keeps footage under nested folders (e.g., `/final/video.mp4`) gets `has_heavy_files: false`, which downstream sync logic relies on.  
+- **Sync exclusion filter misidentifies generated folders:** `EXCLUDE_PATTERNS` contain glob syntax (`**/node_modules/**`, `**/.DS_Store`), but `excluded_file?` strips `**/` and compares raw path segments (`lib/appydave/tools/dam/sync_from_ssd.rb:160-182`), so patterns like `.DS_Store` or `.turbo` may still slip through or block unrelated files. Consider using `File.fnmatch` with the original glob rather than manual string surgery.
+
+Action: Fold these findings into the existing VAT manifest backlog or spin up DAM‑specific tickets so both manifest implementations converge on a single, tested service.
+
 ---
 
 ### ⚠️ CLI Standardization (Worth Auditing)

data/lib/appydave/tools/dam/manifest_generator.rb

@@ -86,12 +86,23 @@ module Appydave
 
           # Scan SSD (if available)
           if ssd_available
-            Dir.glob(File.join(ssd_backup, '*/')).each do |project_path|
-              all_project_ids << File.basename(project_path)
+            Dir.glob(File.join(ssd_backup, '*/')).each do |ssd_path|
+              basename = File.basename(ssd_path)
+
+              if range_folder?(basename)
+                # Scan projects within SSD range folders
+                Dir.glob(File.join(ssd_path, '*/')).each do |project_path|
+                  project_id = File.basename(project_path)
+                  all_project_ids << project_id if valid_project_id?(project_id)
+                end
+              elsif valid_project_id?(basename)
+                # Direct project in SSD root (legacy structure)
+                all_project_ids << basename
+              end
             end
           end
 
-          # Scan local (all projects in brand directory)
+          # Scan local flat structure (active projects only)
           Dir.glob(File.join(brand_path, '*/')).each do |path|
             basename = File.basename(path)
             # Skip hidden and special directories
@@ -101,36 +112,75 @@ module Appydave
             all_project_ids << basename if valid_project_id?(basename)
           end
 
+          # Scan archived structure (restored/archived projects)
+          archived_base = File.join(brand_path, 'archived')
+          if Dir.exist?(archived_base)
+            # Scan range folders (e.g., archived/a50-a99/, archived/b50-b99/)
+            Dir.glob(File.join(archived_base, '*/')).each do |range_folder|
+              # Scan projects within each range folder
+              Dir.glob(File.join(range_folder, '*/')).each do |project_path|
+                basename = File.basename(project_path)
+                all_project_ids << basename if valid_project_id?(basename)
+              end
+            end
+          end
+
           all_project_ids.uniq.sort
         end
 
         def build_project_entries(all_project_ids, ssd_backup, ssd_available)
-          projects = []
-
-          all_project_ids.each do |project_id|
-            local_path = File.join(brand_path, project_id)
-            ssd_path = ssd_available ? File.join(ssd_backup, project_id) : nil
-
-            local_exists = Dir.exist?(local_path)
-            ssd_exists = ssd_path && Dir.exist?(ssd_path)
-
-            projects << {
-              id: project_id,
-              storage: {
-                ssd: {
-                  exists: ssd_exists,
-                  path: ssd_exists ? project_id : nil
-                },
-                local: {
-                  exists: local_exists,
-                  has_heavy_files: local_exists ? heavy_files?(local_path) : false,
-                  has_light_files: local_exists ? light_files?(local_path) : false
-                }
+          all_project_ids.map { |project_id| build_project_entry(project_id, ssd_backup, ssd_available) }
+        end
+
+        def build_project_entry(project_id, ssd_backup, ssd_available)
+          # Check flat structure (active projects)
+          flat_path = File.join(brand_path, project_id)
+          flat_exists = Dir.exist?(flat_path)
+
+          # Check archived structure (restored/archived projects)
+          range = determine_range(project_id)
+          archived_path = File.join(brand_path, 'archived', range, project_id)
+          archived_exists = Dir.exist?(archived_path)
+
+          # Determine which path to use for file detection
+          local_path = if flat_exists
+                         flat_path
+                       else
+                         (archived_exists ? archived_path : flat_path)
+                       end
+          local_exists = flat_exists || archived_exists
+
+          # Determine structure type
+          structure = if flat_exists
+                        'flat'
+                      elsif archived_exists
+                        'archived'
+                      end
+
+          # Check SSD (try both flat and range-based structures)
+          ssd_exists = if ssd_available
+                         flat_ssd_path = File.join(ssd_backup, project_id)
+                         range_ssd_path = File.join(ssd_backup, range, project_id)
+                         Dir.exist?(flat_ssd_path) || Dir.exist?(range_ssd_path)
+                       else
+                         false
+                       end
+
+          {
+            id: project_id,
+            storage: {
+              ssd: {
+                exists: ssd_exists,
+                path: ssd_exists ? project_id : nil
+              },
+              local: {
+                exists: local_exists,
+                structure: structure,
+                has_heavy_files: local_exists ? heavy_files?(local_path) : false,
+                has_light_files: local_exists ? light_files?(local_path) : false
               }
             }
-          end
-
-          projects
+          }
         end
 
         def calculate_disk_usage(projects, ssd_backup)
@@ -139,13 +189,27 @@ module Appydave
 
           projects.each do |project|
             if project[:storage][:local][:exists]
-              local_path = File.join(brand_path, project[:id])
-              local_bytes += calculate_directory_size(local_path)
+              # Try flat structure first, then archived structure
+              flat_path = File.join(brand_path, project[:id])
+              if Dir.exist?(flat_path)
+                local_bytes += calculate_directory_size(flat_path)
+              else
+                range = determine_range(project[:id])
+                archived_path = File.join(brand_path, 'archived', range, project[:id])
+                local_bytes += calculate_directory_size(archived_path) if Dir.exist?(archived_path)
+              end
             end
 
-            if project[:storage][:ssd][:exists]
-              ssd_path = File.join(ssd_backup, project[:id])
-              ssd_bytes += calculate_directory_size(ssd_path)
+            next unless project[:storage][:ssd][:exists]
+
+            # Try flat structure first, then range-based structure
+            flat_ssd_path = File.join(ssd_backup, project[:id])
+            if Dir.exist?(flat_ssd_path)
+              ssd_bytes += calculate_directory_size(flat_ssd_path)
+            else
+              range = determine_range(project[:id])
+              range_ssd_path = File.join(ssd_backup, range, project[:id])
+              ssd_bytes += calculate_directory_size(range_ssd_path) if Dir.exist?(range_ssd_path)
             end
           end
 
@@ -190,6 +254,26 @@ module Appydave
         end
 
         # Helper methods
+
+        # Determine range folder for project
+        # Both SSD and local archived use 50-number ranges with letter prefixes:
+        # b00-b49, b50-b99, a01-a49, a50-a99
+        def determine_range(project_id)
+          # FliVideo/Modern pattern: b40, a82, etc.
+          if project_id =~ /^([a-z])(\d+)/
+            letter = Regexp.last_match(1)
+            number = Regexp.last_match(2).to_i
+            # 50-number ranges (0-49, 50-99)
+            range_start = (number / 50) * 50
+            range_end = range_start + 49
+            # Format with leading zeros and letter prefix
+            format("#{letter}%02d-#{letter}%02d", range_start, range_end)
+          else
+            # Legacy pattern or unknown
+            '000-099'
+          end
+        end
+
         def valid_project_id?(project_id)
           # Valid formats:
           # - Modern: letter + 2 digits + dash + name (e.g., b63-flivideo)
@@ -197,6 +281,23 @@ module Appydave
           !!(project_id =~ /^[a-z]\d{2}-/ || project_id =~ /^\d/)
         end
 
+        def range_folder?(folder_name)
+          # Range folder patterns with letter prefixes:
+          # - b00-b49, b50-b99, a00-a49, a50-a99 (letter + 2 digits + dash + same letter + 2 digits)
+          # - 000-099 (3 digits + dash + 3 digits)
+          # Must match: same letter on both sides (b00-b49, not b00-a49)
+          return true if folder_name =~ /^\d{3}-\d{3}$/
+
+          if folder_name =~ /^([a-z])(\d{2})-([a-z])(\d{2})$/
+            letter1 = Regexp.last_match(1)
+            letter2 = Regexp.last_match(3)
+            # Must be same letter on both sides
+            return letter1 == letter2
+          end
+
+          false
+        end
+
         def heavy_files?(dir)
           return false unless Dir.exist?(dir)
 

data/lib/appydave/tools/dam/s3_operations.rb

@@ -12,6 +12,22 @@ module Appydave
       class S3Operations
         attr_reader :brand_info, :brand, :project_id, :brand_path, :s3_client
 
+        # Directory patterns to exclude from archive/upload (generated/installable content)
+        EXCLUDE_PATTERNS = %w[
+          **/node_modules/**
+          **/.git/**
+          **/.next/**
+          **/dist/**
+          **/build/**
+          **/out/**
+          **/.cache/**
+          **/coverage/**
+          **/.turbo/**
+          **/.vercel/**
+          **/tmp/**
+          **/.DS_Store
+        ].freeze
+
         def initialize(brand, project_id, brand_info: nil, brand_path: nil, s3_client: nil)
           @project_id = project_id
 
@@ -37,9 +53,11 @@ module Appydave
           Aws::S3::Client.new(
             credentials: credentials,
             region: brand_info.aws.region,
-            http_wire_trace: false,
-            ssl_verify_peer: true,
-            ssl_ca_bundle: '/etc/ssl/cert.pem' # macOS system certificates
+            http_wire_trace: false
+            # AWS SDK auto-detects SSL certificates on all platforms:
+            # - Windows: Uses Windows Certificate Store
+            # - macOS: Finds system certificates automatically
+            # - Linux: Finds OpenSSL certificates
           )
         end
 
@@ -500,20 +518,23 @@ module Appydave
           end
 
           size = calculate_directory_size(source_dir)
-          puts '📋 Copy to SSD:'
+          puts '📋 Copy to SSD (excluding generated files):'
           puts "   Source: #{source_dir}"
           puts "   Dest:   #{dest_dir}"
           puts "   Size:   #{file_size_human(size)}"
           puts ''
 
           if dry_run
-            puts '   [DRY-RUN] Would copy entire project to SSD'
+            puts '   [DRY-RUN] Would copy project to SSD (excluding node_modules, .git, etc.)'
             return true
           end
 
-          FileUtils.mkdir_p(File.dirname(dest_dir))
-          FileUtils.cp_r(source_dir, dest_dir, preserve: true)
-          puts '   ✅ Copied to SSD'
+          FileUtils.mkdir_p(dest_dir)
+
+          # Copy files with exclusion filtering
+          stats = copy_with_exclusions(source_dir, dest_dir)
+
+          puts "   ✅ Copied to SSD (#{stats[:files]} files, excluded #{stats[:excluded]} generated files)"
 
           true
         rescue StandardError => e
@@ -521,6 +542,47 @@ module Appydave
           false
         end
 
+        # Copy directory contents with exclusion filtering
+        def copy_with_exclusions(source_dir, dest_dir)
+          stats = { files: 0, excluded: 0 }
+
+          Dir.glob(File.join(source_dir, '**', '*'), File::FNM_DOTMATCH).each do |source_path|
+            next if File.directory?(source_path)
+            next if ['.', '..'].include?(File.basename(source_path))
+
+            relative_path = source_path.sub("#{source_dir}/", '')
+
+            if excluded_path?(relative_path)
+              stats[:excluded] += 1
+              next
+            end
+
+            dest_path = File.join(dest_dir, relative_path)
+            FileUtils.mkdir_p(File.dirname(dest_path))
+            FileUtils.cp(source_path, dest_path, preserve: true)
+            stats[:files] += 1
+          end
+
+          stats
+        end
+
+        # Check if path should be excluded (generated/installable content)
+        def excluded_path?(relative_path)
+          EXCLUDE_PATTERNS.any? do |pattern|
+            # Extract directory/file name from pattern (remove **)
+            excluded_name = pattern.gsub('**/', '').chomp('/**')
+            path_segments = relative_path.split('/')
+
+            if excluded_name.include?('*')
+              # Pattern with wildcards - use fnmatch on filename
+              File.fnmatch(excluded_name, File.basename(relative_path))
+            else
+              # Check if any path segment matches the excluded name
+              path_segments.include?(excluded_name)
+            end
+          end
+        end
+
         # Delete local project directory
         def delete_local_project(project_dir, dry_run: false)
           size = calculate_directory_size(project_dir)

data/lib/appydave/tools/dam/sync_from_ssd.rb

@@ -35,6 +35,22 @@ module Appydave
           *.webm
         ].freeze
 
+        # Directory patterns to exclude (generated/installable content)
+        EXCLUDE_PATTERNS = %w[
+          **/node_modules/**
+          **/.git/**
+          **/.next/**
+          **/dist/**
+          **/build/**
+          **/out/**
+          **/.cache/**
+          **/coverage/**
+          **/.turbo/**
+          **/.vercel/**
+          **/tmp/**
+          **/.DS_Store
+        ].freeze
+
         def initialize(brand, brand_info: nil, brand_path: nil)
           @brand_info = brand_info || load_brand_info(brand)
           @brand = @brand_info.key # Use resolved brand key, not original input
@@ -133,11 +149,11 @@ module Appydave
 
         # Determine if project should be synced
         def should_sync_project?(project)
-          # Only sync if project exists on SSD but NOT in local flat structure
+          # Only sync if project exists on SSD but NOT locally (either flat or archived)
           return false unless project[:storage][:ssd][:exists]
 
-          # Skip if exists locally in flat structure
-          return false if project[:storage][:local][:exists] && project[:storage][:local][:structure] == 'flat'
+          # Skip if exists locally in any structure (flat or archived)
+          return false if project[:storage][:local][:exists]
 
           true
         end
@@ -184,6 +200,7 @@ module Appydave
           LIGHT_FILE_PATTERNS.each do |pattern|
             Dir.glob(File.join(ssd_path, pattern)).each do |source_file|
               next if heavy_file?(source_file)
+              next if excluded_file?(source_file, ssd_path)
 
               copy_stats = copy_light_file(source_file, ssd_path, local_dir, dry_run: dry_run)
               stats[:files] += copy_stats[:files]
@@ -199,6 +216,30 @@ module Appydave
           HEAVY_FILE_PATTERNS.any? { |pattern| File.fnmatch(pattern, File.basename(source_file)) }
         end
 
+        # Check if file should be excluded (generated/installable content)
+        def excluded_file?(source_file, ssd_path)
+          relative_path = source_file.sub("#{ssd_path}/", '')
+
+          EXCLUDE_PATTERNS.any? do |pattern|
+            # Extract directory/file name from pattern (remove **)
+            # **/node_modules/** → node_modules
+            # **/.git/** → .git
+            # **/.DS_Store → .DS_Store
+            excluded_name = pattern.gsub('**/', '').chomp('/**')
+
+            # Check path segments for matches
+            path_segments = relative_path.split('/')
+
+            if excluded_name.include?('*')
+              # Pattern with wildcards - use fnmatch on filename
+              File.fnmatch(excluded_name, File.basename(relative_path))
+            else
+              # Check if any path segment matches the excluded name
+              path_segments.include?(excluded_name)
+            end
+          end
+        end
+
         # Copy a single light file
         def copy_light_file(source_file, ssd_path, local_dir, dry_run: false)
           relative_path = source_file.sub("#{ssd_path}/", '')

data/lib/appydave/tools/version.rb

@@ -2,6 +2,6 @@
 
 module Appydave
   module Tools
-    VERSION = '0.18.1'
+    VERSION = '0.18.3'
   end
 end

data/package.json

@@ -1,6 +1,6 @@
 {
   "name": "appydave-tools",
-  "version": "0.18.1",
+  "version": "0.18.3",
   "description": "AppyDave YouTube Automation Tools",
   "scripts": {
     "release": "semantic-release"

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: appydave-tools
 version: !ruby/object:Gem::Version
-  version: 0.18.1
+  version: 0.18.3
 platform: ruby
 authors:
 - David Cruwys
@@ -227,6 +227,10 @@ files:
 - bin/youtube_automation.rb
 - bin/youtube_manager.rb
 - docs/README.md
+- docs/SESSION-SUMMARY-WINDOWS-PREP.md
+- docs/WINDOWS-COMPATIBILITY-REPORT.md
+- docs/WINDOWS-SETUP.md
+- docs/WINDOWS-START-HERE.md
 - docs/archive/codebase-audit-2025-01.md
 - docs/archive/documentation-framework-proposal.md
 - docs/archive/purpose-and-philosophy.md
@@ -241,6 +245,7 @@ files:
 - docs/dam/session-summary-2025-11-09.md
 - docs/dam/usage.md
 - docs/dam/vat-testing-plan.md
+- docs/dam/windows-testing-guide.md
 - docs/development/CODEX-recommendations.md
 - docs/development/README.md
 - docs/development/cli-architecture-patterns.md