appydave-tools 0.18.2 → 0.18.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 754b54be8ce681f9743cadf14ecbd489aaaab2f569ebc78a7a9d70b673f9c8a7
-  data.tar.gz: fd44964465106177b86c17e34e15d125c7256bd7575be16a83182d1f14fbc7d8
+  metadata.gz: 13697577cf21ab548b3ab152ad3a12194cf26c034eeaf2fdf5325b49730fe120
+  data.tar.gz: b1d7ee7daf7b60c00b80b0ec399ca7d5664d38d41cdacec959f8f8ba3f440d64
 SHA512:
-  metadata.gz: 05a625b643fcc72599156d2549459c5c9c800ab1ebd9da11aa82a6e4e4e2fee87e2784bdf610904a8a4b50790a5a7f53d41c2ddc813ea1e98b8f80fcc00da387
-  data.tar.gz: b8d5ab9617a92f586e3a0d4938593b0f92443af7076d0f1d2c854535ba1d68bb932896944557bd5b94074d0bb6d69b49e72a5fd9ae7c05e64919bbc404d0daf1
+  metadata.gz: 8d6d9643703e76b3946ca0f5923b74972e04b3981a5cd2ca3ebbac315d2d42c98064dc5e6dc12dbc0833073e1b7b36c5b41f1276b38acb5a01dd86802f7aabce
+  data.tar.gz: 390852362fe0ecc9c45bf4c45bfb87ef8fc697051026b89e3f93eacdfd55a8ed9796f2892870e2d661ab4491fc0183ac07d4046ae6051588fe4111b1b0140c21

data/CHANGELOG.md

@@ -1,3 +1,18 @@
+## [0.18.3](https://github.com/appydave/appydave-tools/compare/v0.18.2...v0.18.3) (2025-11-10)
+
+
+### Bug Fixes
+
+* resolve archived structure detection and range folder calculation for DAM manifest and sync ([dec1400](https://github.com/appydave/appydave-tools/commit/dec1400c561f11959fb6aecd7761e916ca525082))
+* resolve rubocop violations in manifest_generator (refactor build_project_entry, simplify SSD check) ([ced61be](https://github.com/appydave/appydave-tools/commit/ced61be41c3da352831d46e3486968f5ee55842c))
+
+## [0.18.2](https://github.com/appydave/appydave-tools/compare/v0.18.1...v0.18.2) (2025-11-10)
+
+
+### Bug Fixes
+
+* resolve Windows compatibility by removing hardcoded SSL certificate path ([fb4e6f7](https://github.com/appydave/appydave-tools/commit/fb4e6f74437e898229e1863f361d8ed4274a4ca6))
+
 ## [0.18.1](https://github.com/appydave/appydave-tools/compare/v0.18.0...v0.18.1) (2025-11-10)
 
 

data/docs/dam/usage.md

@@ -561,17 +561,70 @@ dam s3-down
 dam s3-status
 ```
 
-### Dry-Run Mode
+### Command Safety Features
 
-Preview actions without making changes:
+#### Dry-Run Support
+
+All filesystem-modifying commands support `--dry-run` to preview changes before execution:
+
+| Command | Dry-Run Support | Force Flag Required | What It Previews |
+|---------|----------------|---------------------|------------------|
+| **s3-up** | ✅ Yes | No | Files to upload to S3 |
+| **s3-down** | ✅ Yes | No | Files to download from S3 |
+| **s3-cleanup-remote** | ✅ Yes | **Yes (`--force`)** | S3 files to delete |
+| **s3-cleanup-local** | ✅ Yes | **Yes (`--force`)** | Local s3-staging files to delete |
+| **archive** | ✅ Yes | Optional (`--force` = delete local) | Project to copy to SSD |
+| **sync-ssd** | ✅ Yes | No | Light files to restore from SSD |
+
+#### Read-Only Commands (No Dry-Run Needed)
+
+These commands only read data and don't modify files:
+
+| Command | Type | What It Does |
+|---------|------|--------------|
+| **list** | Read-only | List brands/projects |
+| **manifest** | Generates JSON | Generate `projects.json` manifest |
+| **s3-status** | Read-only | Check sync status |
+| **help** | Read-only | Show help information |
+
+#### Safety Workflow Examples
+
+**Always preview destructive operations first:**
 
 ```bash
+# Preview S3 upload
 dam s3-up appydave b65 --dry-run
-dam s3-down voz boy-baker --dry-run
-dam s3-cleanup-remote aitldr movie-posters --dry-run
-dam s3-cleanup-local appydave b65 --dry-run
+# Review output, then execute
+dam s3-up appydave b65
+
+# Preview S3 cleanup (requires --force)
+dam s3-cleanup-remote appydave b65 --force --dry-run
+# Review output, then execute
+dam s3-cleanup-remote appydave b65 --force
+
+# Preview archive (with or without local deletion)
+dam archive appydave b63 --dry-run              # Copy only
+dam archive appydave b63 --force --dry-run      # Copy + delete local
+# Review output, then execute
+dam archive appydave b63                        # Copy only
+dam archive appydave b63 --force                # Copy + delete local
+
+# Preview SSD sync
+dam sync-ssd appydave --dry-run
+# Review output, then execute
+dam sync-ssd appydave
 ```
 
+#### Force Flag Behavior
+
+Commands requiring `--force` provide extra protection for destructive operations:
+
+- **s3-cleanup-remote**: Must use `--force` to delete S3 files (prevents accidental deletion)
+- **s3-cleanup-local**: Must use `--force` to delete local staging files
+- **archive**: Optional `--force` flag deletes local copy after successful SSD backup
+  - Without `--force`: Copies to SSD, keeps local copy intact
+  - With `--force`: Copies to SSD, then deletes local copy (frees disk space)
+
 ### Interactive Selection
 
 When multiple projects match short name:

data/docs/dam/vat-testing-plan.md

@@ -113,6 +113,62 @@ dam list
 
 ---
 
+## Command Safety Features Reference
+
+### Dry-Run and Force Flag Support
+
+All filesystem-modifying commands support `--dry-run` to preview changes before execution:
+
+| Command | Dry-Run Support | Force Flag Required | What It Does |
+|---------|----------------|---------------------|--------------|
+| **s3-up** | ✅ Yes | No | Preview files to upload to S3 |
+| **s3-down** | ✅ Yes | No | Preview files to download from S3 |
+| **s3-cleanup-remote** | ✅ Yes | **Yes (`--force`)** | Preview S3 files to delete |
+| **s3-cleanup-local** | ✅ Yes | **Yes (`--force`)** | Preview local s3-staging files to delete |
+| **archive** | ✅ Yes | Optional (`--force` = delete local) | Preview project copy to SSD |
+| **sync-ssd** | ✅ Yes | No | Preview light files to restore from SSD |
+
+### Read-Only Commands (No Dry-Run Needed)
+
+These commands only read data and don't modify files:
+
+| Command | Type | What It Does |
+|---------|------|--------------|
+| **list** | Read-only | List brands/projects |
+| **manifest** | Generates JSON | Generate `projects.json` manifest |
+| **s3-status** | Read-only | Check sync status |
+| **help** | Read-only | Show help information |
+
+### Force Flag Behavior
+
+Commands requiring `--force` provide extra protection for destructive operations:
+
+- **s3-cleanup-remote**: Must use `--force` to delete S3 files (prevents accidental deletion)
+- **s3-cleanup-local**: Must use `--force` to delete local staging files
+- **archive**: Optional `--force` flag deletes local copy after successful SSD backup
+  - Without `--force`: Copies to SSD, keeps local copy intact
+  - With `--force`: Copies to SSD, then deletes local copy (frees disk space)
+
+### Safety Testing Workflow
+
+**Always test with dry-run first:**
+
+```bash
+# 1. Preview with dry-run
+dam s3-up appydave b65 --dry-run
+dam s3-cleanup-remote appydave b65 --force --dry-run
+dam archive appydave b63 --dry-run
+
+# 2. Review output carefully
+
+# 3. Execute if safe
+dam s3-up appydave b65
+dam s3-cleanup-remote appydave b65 --force
+dam archive appydave b63
+```
+
+---
+
 ## Test Suite
 
 ### Phase 1: Unit Tests (Automated - RSpec)

data/docs/development/CODEX-recommendations.md

@@ -1,6 +1,6 @@
 # CODEX Recommendations - Review & Status
 
-> Last updated: 2025-11-10
+> Last updated: 2025-11-10 08:12:59 UTC
 > Original recommendations provided by Codex (GPT-5) on 2025-11-09
 
 This document captures Codex's architectural recommendations with implementation status and verdicts after engineering review.
@@ -146,6 +146,19 @@ end
 
 **Note:** These are legitimate technical debt items, not style preferences. Recommend creating GitHub issues for tracking.
 
+### 🔍 DAM Manifest & Sync Addendum (2025-11-10)
+
+New DAM code mirrors the VAT manifest/sync flow but reintroduces several bugs plus new inconsistencies:
+
+- **Archived projects still missing:** `collect_project_ids` explicitly skips the `archived` directory (`lib/appydave/tools/dam/manifest_generator.rb:70-99`), so the later logic that probes `brand_path/archived/<range>/<project>` never runs; manifests omit the majority of historical work. This also means `SyncFromSsd.should_sync_project?` (`lib/appydave/tools/dam/sync_from_ssd.rb:77-96`) will think everything is already local because manifests never flag archived presence.  
+- **Range math diverges between components:** Manifest uses `project_id =~ /^[a-z](\d+)/` to build ranges (`lib/appydave/tools/dam/manifest_generator.rb:214-224`), but the SSD sync hard-codes `/^b(\d+)/` (`lib/appydave/tools/dam/sync_from_ssd.rb:123-138`). Projects outside the `b` prefix (aitldr, voz, etc.) will all collapse into the fallback `000-099`, creating collisions.  
+- **SSD paths lose grouping info:** Manifests record `path: project_id` for SSD entries (`lib/appydave/tools/dam/manifest_generator.rb:119-126`), ignoring the range folders that exist on disk. The sync tool then assumes `ssd/<project_id>` (line 98) and will fail whenever the SSD organizes projects under range subdirectories.  
+- **Disk usage ignores archived location:** Even when a project only exists under `archived/<range>`, `calculate_disk_usage` points at `File.join(brand_path, project[:id])` (`lib/appydave/tools/dam/manifest_generator.rb:131-146`), so archived-only projects report 0 bytes. Need to reuse the resolved `local_path` (flat vs archived) instead of rebuilding the path blindly.  
+- **Heavy file detection still shallow:** `heavy_files?` only inspects direct children (`lib/appydave/tools/dam/manifest_generator.rb:233-239`), while `light_files?` walks `**/*`. Any team that keeps footage under nested folders (e.g., `/final/video.mp4`) gets `has_heavy_files: false`, which downstream sync logic relies on.  
+- **Sync exclusion filter misidentifies generated folders:** `EXCLUDE_PATTERNS` contain glob syntax (`**/node_modules/**`, `**/.DS_Store`), but `excluded_file?` strips `**/` and compares raw path segments (`lib/appydave/tools/dam/sync_from_ssd.rb:160-182`), so patterns like `.DS_Store` or `.turbo` may still slip through or block unrelated files. Consider using `File.fnmatch` with the original glob rather than manual string surgery.
+
+Action: Fold these findings into the existing VAT manifest backlog or spin up DAM‑specific tickets so both manifest implementations converge on a single, tested service.
+
 ---
 
 ### ⚠️ CLI Standardization (Worth Auditing)

data/lib/appydave/tools/dam/manifest_generator.rb

@@ -86,12 +86,23 @@ module Appydave
 
           # Scan SSD (if available)
           if ssd_available
-            Dir.glob(File.join(ssd_backup, '*/')).each do |project_path|
-              all_project_ids << File.basename(project_path)
+            Dir.glob(File.join(ssd_backup, '*/')).each do |ssd_path|
+              basename = File.basename(ssd_path)
+
+              if range_folder?(basename)
+                # Scan projects within SSD range folders
+                Dir.glob(File.join(ssd_path, '*/')).each do |project_path|
+                  project_id = File.basename(project_path)
+                  all_project_ids << project_id if valid_project_id?(project_id)
+                end
+              elsif valid_project_id?(basename)
+                # Direct project in SSD root (legacy structure)
+                all_project_ids << basename
+              end
             end
           end
 
-          # Scan local (all projects in brand directory)
+          # Scan local flat structure (active projects only)
           Dir.glob(File.join(brand_path, '*/')).each do |path|
             basename = File.basename(path)
             # Skip hidden and special directories
@@ -101,36 +112,75 @@ module Appydave
             all_project_ids << basename if valid_project_id?(basename)
           end
 
+          # Scan archived structure (restored/archived projects)
+          archived_base = File.join(brand_path, 'archived')
+          if Dir.exist?(archived_base)
+            # Scan range folders (e.g., archived/a50-a99/, archived/b50-b99/)
+            Dir.glob(File.join(archived_base, '*/')).each do |range_folder|
+              # Scan projects within each range folder
+              Dir.glob(File.join(range_folder, '*/')).each do |project_path|
+                basename = File.basename(project_path)
+                all_project_ids << basename if valid_project_id?(basename)
+              end
+            end
+          end
+
           all_project_ids.uniq.sort
         end
 
         def build_project_entries(all_project_ids, ssd_backup, ssd_available)
-          projects = []
-
-          all_project_ids.each do |project_id|
-            local_path = File.join(brand_path, project_id)
-            ssd_path = ssd_available ? File.join(ssd_backup, project_id) : nil
-
-            local_exists = Dir.exist?(local_path)
-            ssd_exists = ssd_path && Dir.exist?(ssd_path)
-
-            projects << {
-              id: project_id,
-              storage: {
-                ssd: {
-                  exists: ssd_exists,
-                  path: ssd_exists ? project_id : nil
-                },
-                local: {
-                  exists: local_exists,
-                  has_heavy_files: local_exists ? heavy_files?(local_path) : false,
-                  has_light_files: local_exists ? light_files?(local_path) : false
-                }
+          all_project_ids.map { |project_id| build_project_entry(project_id, ssd_backup, ssd_available) }
+        end
+
+        def build_project_entry(project_id, ssd_backup, ssd_available)
+          # Check flat structure (active projects)
+          flat_path = File.join(brand_path, project_id)
+          flat_exists = Dir.exist?(flat_path)
+
+          # Check archived structure (restored/archived projects)
+          range = determine_range(project_id)
+          archived_path = File.join(brand_path, 'archived', range, project_id)
+          archived_exists = Dir.exist?(archived_path)
+
+          # Determine which path to use for file detection
+          local_path = if flat_exists
+                         flat_path
+                       else
+                         (archived_exists ? archived_path : flat_path)
+                       end
+          local_exists = flat_exists || archived_exists
+
+          # Determine structure type
+          structure = if flat_exists
+                        'flat'
+                      elsif archived_exists
+                        'archived'
+                      end
+
+          # Check SSD (try both flat and range-based structures)
+          ssd_exists = if ssd_available
+                         flat_ssd_path = File.join(ssd_backup, project_id)
+                         range_ssd_path = File.join(ssd_backup, range, project_id)
+                         Dir.exist?(flat_ssd_path) || Dir.exist?(range_ssd_path)
+                       else
+                         false
+                       end
+
+          {
+            id: project_id,
+            storage: {
+              ssd: {
+                exists: ssd_exists,
+                path: ssd_exists ? project_id : nil
+              },
+              local: {
+                exists: local_exists,
+                structure: structure,
+                has_heavy_files: local_exists ? heavy_files?(local_path) : false,
+                has_light_files: local_exists ? light_files?(local_path) : false
               }
             }
-          end
-
-          projects
+          }
         end
 
         def calculate_disk_usage(projects, ssd_backup)
@@ -139,13 +189,27 @@ module Appydave
 
           projects.each do |project|
             if project[:storage][:local][:exists]
-              local_path = File.join(brand_path, project[:id])
-              local_bytes += calculate_directory_size(local_path)
+              # Try flat structure first, then archived structure
+              flat_path = File.join(brand_path, project[:id])
+              if Dir.exist?(flat_path)
+                local_bytes += calculate_directory_size(flat_path)
+              else
+                range = determine_range(project[:id])
+                archived_path = File.join(brand_path, 'archived', range, project[:id])
+                local_bytes += calculate_directory_size(archived_path) if Dir.exist?(archived_path)
+              end
             end
 
-            if project[:storage][:ssd][:exists]
-              ssd_path = File.join(ssd_backup, project[:id])
-              ssd_bytes += calculate_directory_size(ssd_path)
+            next unless project[:storage][:ssd][:exists]
+
+            # Try flat structure first, then range-based structure
+            flat_ssd_path = File.join(ssd_backup, project[:id])
+            if Dir.exist?(flat_ssd_path)
+              ssd_bytes += calculate_directory_size(flat_ssd_path)
+            else
+              range = determine_range(project[:id])
+              range_ssd_path = File.join(ssd_backup, range, project[:id])
+              ssd_bytes += calculate_directory_size(range_ssd_path) if Dir.exist?(range_ssd_path)
             end
           end
 
@@ -190,6 +254,26 @@ module Appydave
         end
 
         # Helper methods
+
+        # Determine range folder for project
+        # Both SSD and local archived use 50-number ranges with letter prefixes:
+        # b00-b49, b50-b99, a01-a49, a50-a99
+        def determine_range(project_id)
+          # FliVideo/Modern pattern: b40, a82, etc.
+          if project_id =~ /^([a-z])(\d+)/
+            letter = Regexp.last_match(1)
+            number = Regexp.last_match(2).to_i
+            # 50-number ranges (0-49, 50-99)
+            range_start = (number / 50) * 50
+            range_end = range_start + 49
+            # Format with leading zeros and letter prefix
+            format("#{letter}%02d-#{letter}%02d", range_start, range_end)
+          else
+            # Legacy pattern or unknown
+            '000-099'
+          end
+        end
+
         def valid_project_id?(project_id)
           # Valid formats:
           # - Modern: letter + 2 digits + dash + name (e.g., b63-flivideo)
@@ -197,6 +281,23 @@ module Appydave
           !!(project_id =~ /^[a-z]\d{2}-/ || project_id =~ /^\d/)
         end
 
+        def range_folder?(folder_name)
+          # Range folder patterns with letter prefixes:
+          # - b00-b49, b50-b99, a00-a49, a50-a99 (letter + 2 digits + dash + same letter + 2 digits)
+          # - 000-099 (3 digits + dash + 3 digits)
+          # Must match: same letter on both sides (b00-b49, not b00-a49)
+          return true if folder_name =~ /^\d{3}-\d{3}$/
+
+          if folder_name =~ /^([a-z])(\d{2})-([a-z])(\d{2})$/
+            letter1 = Regexp.last_match(1)
+            letter2 = Regexp.last_match(3)
+            # Must be same letter on both sides
+            return letter1 == letter2
+          end
+
+          false
+        end
+
         def heavy_files?(dir)
           return false unless Dir.exist?(dir)
 

data/lib/appydave/tools/dam/sync_from_ssd.rb

@@ -149,11 +149,11 @@ module Appydave
 
         # Determine if project should be synced
         def should_sync_project?(project)
-          # Only sync if project exists on SSD but NOT in local flat structure
+          # Only sync if project exists on SSD but NOT locally (either flat or archived)
           return false unless project[:storage][:ssd][:exists]
 
-          # Skip if exists locally in flat structure
-          return false if project[:storage][:local][:exists] && project[:storage][:local][:structure] == 'flat'
+          # Skip if exists locally in any structure (flat or archived)
+          return false if project[:storage][:local][:exists]
 
           true
         end

data/lib/appydave/tools/version.rb

@@ -2,6 +2,6 @@
 
 module Appydave
   module Tools
-    VERSION = '0.18.2'
+    VERSION = '0.18.4'
   end
 end

data/package.json

@@ -1,6 +1,6 @@
 {
   "name": "appydave-tools",
-  "version": "0.18.2",
+  "version": "0.18.4",
   "description": "AppyDave YouTube Automation Tools",
   "scripts": {
     "release": "semantic-release"

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: appydave-tools
 version: !ruby/object:Gem::Version
-  version: 0.18.2
+  version: 0.18.4
 platform: ruby
 authors:
 - David Cruwys