storazzo 0.6.1 → 0.7.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: e82c96fbdff7b8d2f78556899257b431ec1e58ec255321bae2afb4ba3b1ac309
-  data.tar.gz: f343d195bce5121d18f456b3fd90f8032a41427ae41e4248f5b8628aca1b0a3e
+  metadata.gz: 6948ac6219b8540a46f7a2c0a155e097168656ed6b09f0adc9b2601b62bc466d
+  data.tar.gz: b6f40f0dd0c025c0a03b59a11ba75b9a6ed0f398f93fb6ce1bf18538ef9ebdbf
 SHA512:
-  metadata.gz: d9aa08e7fcc62e0d8a3510db4d26f27e227aa778f8e9d8e6d7831676936c282e97f72f57f08febdc7383dec2f1e4c66beb42eb94dae3638f422fc6073292f204
-  data.tar.gz: e75cb487f55943dbae3d23242909ca61a83c39aaf5ef793225c074181f6914290d7f11aa3fa537f43010db4feab239efb14e88c5f30d91f737465081ad825309
+  metadata.gz: e2133288626b087aa3ec75c95626b7f23cb09b3073589bef4582837b12f7a1559b96afc0a7d42c4422c9fae61372143673170fcfca6ad6bfc9a509b3f1cef404
+  data.tar.gz: 1dc88ed96a2d77e5b38f68528315cecc409263bd350acb657ed643eb8e451247db28a23319816f2cfe3a65dc81b43a04fb90b059be36e6e289507a95b23ce49f

data/Gemfile

@@ -2,6 +2,8 @@
 
 source 'https://rubygems.org'
 
+gemspec
+
 # to troubleshoot/debug things
 gem 'rake'
 gem 'rubocop'
@@ -9,4 +11,17 @@ gem 'ruby-lsp', '~> 0.0.4', group: :development
 
 # gem 'pry'
 gem 'pry', group: :development
-# gem 'after_do' # To allow AFTER calling instance to do LOAD() automagically. Naah I can do without :)
+gem 'minitest', group: :development
+
+gem 'sqlite3'
+gem 'thor'
+gem 'google-cloud-storage'
+
+group :test do
+  gem 'minitest'
+  gem 'minitest-reporters'
+  gem 'mocha'
+  gem 'rspec'
+  gem 'cucumber'
+  gem 'aruba' # For testing CLI applications
+end

data/README.md

@@ -3,35 +3,34 @@
 
 📦 Storazzo 💎 gem - a Gem to automatically parse your FS for mounts (💽 💾 💿 ) and compute MD5 (🤐) of all files therein and then collect in central DB 🔋 through 📦 StorazzoApp📦 (TM).
 
+![Storazzo logo](assets/storazzo-logo.png)
+
 # INSTALL
 
 `gem install storazzo`
 
 (Latest version is hosted in https://rubygems.org/gems/storazzo)
 
-# Tests
-
-I still struggle to enforce the include of LOCAL unchecked code rather than latest required system gem (cmon Ruby!)
-but I found loads of interesting ways to test my code by googling and StoackOverflowing:
-
-* `rake test TEST="test/sum_test.rb"`
-* test-gcs-bucket: `ruby -I test test/test_gcs_bucket.rb` (meh - see below)
-* test-media-subfolder: `rake test TEST="test/media/*.rb"`
+# Development & Testing
 
-Single test in single file:
+To run the tools locally without installing the gem, use `just` (recommended) or standard Ruby flags.
 
-* `rake test TEST="test/sum_test.rb" TESTOPTS="--name=test_returns_two"` (sample)
-* `rake test TEST="test/media/test_local_folder.rb" TESTOPTS="--name=test_1_first_directory_parsing_actually_works"`
-* `ruby -I test test/test_local_folder.rb -n test_first_directory_parsing_actually_works` (note this includes `storazzo` latest gem 
-    and doesnt benefit from LATEST code so its NOT good for testing: use RAKE for that).
+### Using `just` (Recommended)
+This repository includes a `justfile` with common development commands:
+* `just setup` - Install dependencies
+* `just test` - Run all tests
+* `just run scan /path` - Run the local `storazzo` CLI
+* `just hello` - Run the local `hello-storazzo` script
 
-**Testing binary files** is hard: by default they 'require storazzo' and use the GEM INSTALLed version which is a few versions away, usually.
-So while developing you might want to include the lib/ folder, like this:
+### Manual Local Execution
+If you don't have `just` installed, you can use `bundle exec` and the `-Ilib` flag:
+* **Run CLI**: `bundle exec ruby -Ilib bin/storazzo scan /path`
+* **Run Tests**: `bundle exec rake test`
+* **Single Test**: `bundle exec ruby -Ilib:test test/media/test_local_folder.rb`
 
-* Use local gem (super latest) for checking latest code: `ruby -Ilib bin/hello-storazzo`
-* This will use the gem installed a few days ago, likely so wont do you any good to test latest code: `bin/hello-storazzo`
+### Why `bundle exec` and `-Ilib`?
+Using `bundle exec` ensures all dependencies from the `Gemfile` are available. The `-Ilib` flag adds the local `lib/` directory to the Ruby load path, ensuring your local changes are used instead of any installed gem version.
 
-Now to toggle verbosity I believe I need to go into Rakefile (bummer)
 # Thanks
 
 Inspiration from:

data/VERSION

@@ -1 +1 @@
-0.6.1
+0.7.3

data/bin/hello-storazzo

@@ -1,6 +1,10 @@
 #!/usr/bin/env ruby
 # frozen_string_literal: true
 
+require 'bundler/setup'
+$LOAD_PATH.unshift(File.expand_path('../lib', __dir__))
 require 'storazzo'
 
-Storazzo::Main.say_hi(ARGV[0])
+msg = ARGV.empty? ? 'HelloStorazzo' : ARGV.join(' ')
+
+Storazzo::Main.say_hi(msg)

data/bin/ricdisk-magic

@@ -1,21 +1,15 @@
 #!/usr/bin/env ruby
 # frozen_string_literal: true
 
-############# ############# ############# ############# ############# ############# ############# #############
-# placeholder until i make it work..
-# https://guides.rubygems.org/make-your-own-gem/#adding-an-executable
-# require 'storazzo'
-# require_relative '../lib/storazzo'
+require 'bundler/setup'
+$LOAD_PATH.unshift(File.expand_path('../lib', __dir__))
+require 'storazzo'
 require 'fileutils'
 require 'yaml'
 require 'socket'
-require 'optparse' # http://ruby.about.com/od/advancedruby/a/optionparser.htm
+require 'optparse'
 
-# Including local gem.
-# puts File.expand_path(FileUtils.pwd, "/../")
-# local_gem_path = File.expand_path(FileUtils.pwd, "/../")
-# gem 'storazzo', path: local_gem_path
-require 'storazzo'
+# require 'storazzo'
 # include Storazzo
 # include Storazzo::Colors
 extend Storazzo::Colors
@@ -32,8 +26,8 @@ include Storazzo::Common # instead
 # include 'lib/ric_disk'
 # extend Storazzo::Colors
 
-if RUBY_VERSION.split('.')[0] == 1
-  puts 'Refusing to launch a script form Ruby 1. Sorry Ric, its 2020 damn it!'
+if RUBY_VERSION.split('.')[0].to_i < 3
+  puts 'Refusing to launch a script form Ruby 1 or 2. Sorry Ric, its 2020 damn it!'
   exit 2020
 end
 

data/bin/stats-with-md5

@@ -7,7 +7,7 @@ require 'optparse' # http://ruby.about.com/od/advancedruby/a/optionparser.htm
 require 'date' # for DateTime
 require 'tempfile'
 
-if RUBY_VERSION.split('.')[0] == 1
+if RUBY_VERSION.split('.')[0].to_i == 1
   puts 'Refusing to launch a script form Ruby 1. Sorry Ric, its 2020 damn it!'
   exit 2020
 end

data/bin/storazzo

@@ -1,158 +1,112 @@
 #!/usr/bin/env ruby
 # frozen_string_literal: true
 
+require 'bundler/setup'
+$LOAD_PATH.unshift(File.expand_path('../lib', __dir__))
 require 'storazzo'
-# gem 'storazzo', :path => File.expand_path("../lib", __FILE__)
+require 'thor'
 
-require 'optparse'
-
-extend Storazzo::Colors
-include Storazzo::Common # instead
-
-if RUBY_VERSION.split('.')[0] == 1
-  puts 'Refusing to launch a script form Ruby 1. Sorry Ric, its 2020 damn it!'
+if RUBY_VERSION.split('.')[0].to_i < 3
+  puts 'Refusing to launch a script form Ruby < 3.'
   exit 2020
 end
 
-$PROG_VER = '0.1'
-$DEBUG    = ENV['DEBUG'] == 'true' # (true/false)
-
-$actions = %w[
-  show auto help
-].sort
-
-# you can't but love Ruby monkey-patching ability..
-class Array
-  #  puts "array drop"
-  def tail
-    drop(1) # https://stackoverflow.com/questions/3615700/ruby-what-is-the-easiest-way-to-remove-the-first-element-from-an-array
-  end
-end
+module Storazzo
+  class CLI < Thor
+    include ::Storazzo::Colors
+    include ::Storazzo::Common
+
+    class_option :verbose, type: :boolean, default: false, desc: 'Output more information'
+    class_option :debug, type: :boolean, default: false, desc: 'Enable debug mode'
+
+    desc "scan PATH", "Scan a removable disk or local folder and upload its metadata to GCS"
+    def scan(path)
+      setup_env
+      puts "Scanning path: #{path}"
+      
+      dir = File.expand_path(path)
+      unless File.directory?(dir)
+        puts "Error: #{dir} is not a valid directory."
+        exit 1
+      end
+
+      config = Storazzo::RicDiskConfig.instance
+      config.load
+
+      rd = Storazzo::RicDisk.new(Storazzo::Media::AbstractRicDisk.DirFactory(dir))
+      rd.write_config_yaml_to_disk(dir)
+      rd.compute_stats_files(upload_to_gcs: true)
+
+      puts "Scan completed for #{dir}."
+    end
 
-HISTORY = <<-BIG_LONG_MULTILINE
-    2022-0806  v0.1 First stesure
-BIG_LONG_MULTILINE
-
-$description = <<DESCRIPTION_HEREFILE.gsub(/\s+/, ' ').strip
-    This util shall expose all Storazzo aPIs via CLI: show-local-dirs, ...
-    show GCS buckets, ...
-
-    It's also my dream to add some nice icons which are joy to the eye, like
-    some  [ ] [x] before the list to show if its agood or bad directory and type also#{' '}
-DESCRIPTION_HEREFILE
-# >> "SELECT * FROM users ORDER BY users.id DESC"
-
-$myconf = {
-  app_name: "Storazzo CLI: #{$PROGRAM_NAME}",
-  description: $description
-  # TODO: move to some class default
-  # :media_dirs => %w{/media/riccardo/ /Volumes/ /mnt/ ~/git/storazzo/var/test/
-  #                  /sobenme/giusto/per/imparare/ad/ammutolire/gli/errori/},
-  # :mount_types => %w{vfat ntfs},
-}
-
-def usage(comment = nil)
-  puts white($optparse.banner)
-  puts($optparse.summarize)
-  # puts("Description: " + gray($myconf[:description]))
-  puts red("ERROR: #{comment}") if comment
-  # puts "Description: #{ $myconf[:description] }"
-  exit 13
-end
+    desc "search QUERY", "Search for a keyword across all indexed disks via SQLite DB"
+    def search(query)
+      setup_env
+      require 'storazzo/search_engine'
+      
+      puts "Searching for: #{query}"
+      engine = Storazzo::SearchEngine.new
+      engine.sync_from_gcs
+      results = engine.query(query)
+      
+      if results.empty?
+        puts "No results found for '#{query}'."
+      else
+        puts "Found #{results.size} matches:"
+        results.each do |r|
+          puts "- [#{r[:disk]}] #{r[:path]} (size: #{r[:size]})"
+        end
+      end
+    end
 
-# include it in main if you want a custome one
-# see lib_autoinit in lib/util.rb
-def init
-  $opts = {}
-  # setting defaults
-  $opts[:verbose] = false
-  $opts[:dryrun] = false
-  $opts[:debug] = false
-  $opts[:force] = false
-
-  $optparse = OptionParser.new do |opts|
-    opts.banner = "#{$PROGRAM_NAME} v.#{$PROG_VER}\n Usage: #{File.basename $PROGRAM_NAME} [options] ACTION [file1] [file2] ..."
-    opts.separator "Actions: #{$actions.join ', '}"
-    opts.separator ' $0 show # show options and local folders '
-    opts.separator ' $0 auto # takes automated actions based on config and local disk. For lazy users'
-    opts.separator 'Options: '
-    opts.on('-d', '--debug', 'enables debug (DFLT=false)') do
-      $opts[:debug] = true
-      $DEBUG = true
+    desc "backup SRC DEST", "Backup a local folder to GCS and update index"
+    def backup(src, dest)
+      setup_env
+      puts "Backing up #{src} to #{dest}"
+      
+      src_dir = File.expand_path(src)
+      unless File.directory?(src_dir)
+        puts "Error: #{src_dir} is not a valid directory."
+        exit 1
+      end
+
+      # 1. Run gsutil rsync
+      cmd = "gsutil -m rsync -r '#{src_dir}' '#{dest}'"
+      puts "Running: #{cmd}"
+      system(cmd)
+      unless $?.success?
+        puts "Failed to backup files to GCS."
+        exit 1
+      end
+
+      # 2. Compute stats on the destination bucket
+      require 'storazzo/search_engine'
+      # We could run `stats-with-md5` for GCS here to generate the index
+      # and upload to metadata bucket
+      puts "Indexing GCS bucket: #{dest}"
+      cmd_stats = "stats-with-md5 --autowrite '#{dest}'"
+      system(cmd_stats)
+
+      puts "Backup and indexing completed."
     end
-    opts.on('-f', '--force', 'force stuff (DFLT=false)') { $opts[:force] = true }
-    opts.on('-h', '--help', 'Display this screen') { usage }
-    # opts.on( '-j', '--jabba', 'Activates my Jabber powerful CLI' ) {  $opts[:jabba] = true  }
-    opts.on('-n', '--dryrun', "Don't really execute code") { $opts[:dryrun] = true }
-    opts.on('-l', '--logfile FILE', 'Write log to FILE')    { |file| $opts[:logfile] = file }
-    opts.on('-v', '--verbose', 'Output more information')   { $opts[:verbose] = true }
-    opts.separator("Description: #{gray($myconf[:description])}")
-    # opts.separator " -- "
-  end
-  $optparse.parse!
-end
 
-def show_action(argv)
-  deb "ARGV remaning is: #{argv}"
-  deb('TODO show_action')
-  # puts(white("1. Config"))
-  # Pry::ColorPrinter.pp $config.to_verbose_s
-  puts(white('2. Mounts/Dirs'))
-  Pry::ColorPrinter.pp Storazzo::Media::GcsBucket.list_all_with_type
-  pp Storazzo::Media::AbstractRicDisk.super_duper_list_all_with_type
-end
+    desc "show", "Show options and local folders"
+    def show
+      setup_env
+      puts("1. Mounts/Dirs")
+      require 'pp'
+      pp Storazzo::Media::GcsBucket.list_all_with_type
+      pp Storazzo::Media::AbstractRicDisk.super_duper_list_all_with_type
+    end
 
-def real_program
-  deb("Hello world from a templated '#{yellow $PROGRAM_NAME}'")
-  deb "+ Options are:  #{gray $opts}"
-  deb "+ Depured args: #{azure ARGV}"
-  deb "+ Script-specifig super-cool conf: #{green $prog_conf_d}"
-  deb "+ Your configuration: #{purple $myconf.inspect}"
-
-  # Your code goes here...
-  puts white("Hello world from #{$myconf[:app_name]}!")
-  puts "Description: '''#{white $myconf[:description]}'''"
-  Storazzo::Main.say_hi("Note: if the version of this storazzo is behind local version youre probably using a old library.
-    I still need to learn how to call the binary with local/current lib/: bundle exec stiKazzi?")
-
-  $config = Storazzo::RicDiskConfig.instance
-  $config.load
-  deb "StorazzoConfig: #{$config}"
-  # config.load # auto_sbrodola(ARGV)
-  puts yellow("ARGV: #{ARGV}")
-  # config.iterate_through_file_list_for_disks(ARGV)
-
-  if_deb? { Storazzo::Main.say_hi("ARGV: #{ARGV}") }
-
-  usage('I need at least one argument (storazzo ACTION)') if ARGV == []
-  argv_action = ARGV[0]
-  case argv_action # first argment - action
-  when 'show', 'list'
-    show_action(ARGV.tail)
-  when 'auto'
-    auto_action(ARGV.tail)
-  when 'help'
-    usage("There you go, here's your HELP :)")
-  when String
-    usage("Unknown action1: #{argv_action}. Available actions: #{$actions}")
-  else
-    usage("Unknown action2, really strange!: '#{argv_action}'. Actions: #{$actions}")
-  end
-end
+    private
 
-def main(filename)
-  deb "I'm called by #{white filename}"
-  deb "HISTORY: #{gray HISTORY}"
-  init # Enable this to have command line parsing capabilities!
-  puts white("$DEBUG is #{$DEBUG}. Tu turn on, call me with -d") unless $DEBUG
-  # warn "[warn] template v#{$TEMPLATE_VER }: proviamo il warn che magari depreca il DEB"
-  real_program
-
-  if_deb? do
-    puts 'First I need to figure out how to bring in all the libraries in here..'
-    Storazzo::Main.say_hi("ARGV is: #{ARGV.join ', '}")
-    puts Storazzo.version
+    def setup_env
+      $DEBUG = options[:debug]
+      $opts = { verbose: options[:verbose], debug: options[:debug] }
+    end
   end
 end
 
-main(__FILE__)
+Storazzo::CLI.start(ARGV)

data/bin/storazzo-symlink.rb

@@ -1 +1 @@
-bin/storazzo
+storazzo

data/bin/storazzo-util

@@ -1 +1 @@
-bin/storazzo
+storazzo

data/lib/storazzo/common.rb

@@ -28,7 +28,12 @@
 # 👶 🧒 👦 👧 👩 👨 🧑 🧔 🧓 👴 👵 👤 👥 👪 👫 👬 👭 👱 👳 👲 🧕 👸 🤴 🎅 🤶 disabled
 # 🧏 🦻 🦮 🦯 🦺 🦼 🦽 🦾 🦿 🤵 👮 👷 💁 💂 🕴 🕵 🦸 🦹 🧙 🧚 🧜 🧝 🧞 🧛 🧟 👼 👿 👻 👹 👺 👽 👾 🛸 💀 ☠ 🕱 🧠 🦴 👁 👀 👂 👃 👄 🗢 👅 🦷 🦵 🦶 💭 🗬 🗭 💬 🗨 🗩 🗪 🗫 🗰 🗱 🗮 🗯 🗣 🗤 🗥 🗦 🗧 💦 💧 💢 💫 💤 💨 💥 💪 🗲 🔥 💡 💩 💯 🔟 🔰 🕲
 require_relative 'colors'
-require 'pry'
+# require 'pry'
+begin
+  require 'pry'
+rescue LoadError
+  # Pry not available, no worries
+end
 
 module Storazzo
   module Common
@@ -52,8 +57,8 @@ module Storazzo
       puts "[W⚠️RN] #{azure(s)}"
     end
 
-    def err(_str)
-      puts "[ERR👎] #{red(s)}" # ⛔
+    def err(str)
+      puts "[ERR👎] #{red(str)}" # ⛔
     end
 
     def bug(s)
@@ -68,8 +73,12 @@ module Storazzo
     end
 
     def ppp(complex_object_to_colorize)
-      # TODO: i need to learn to return without printing..
-      Pry::ColorPrinter.pp(complex_object_to_colorize)
+      if defined?(Pry::ColorPrinter)
+        Pry::ColorPrinter.pp(complex_object_to_colorize)
+      else
+        require 'pp'
+        pp complex_object_to_colorize
+      end
     end
 
     def fatal(s)
@@ -84,6 +93,10 @@ module Storazzo
       `uname`.chomp == 'Linux'
     end
 
+    def slugify(string)
+      string.to_s.downcase.strip.gsub(/\s+/, '_').gsub(/[^\w-]/, '')
+    end
+
     private
 
     def _debug_true

data/lib/storazzo/gcs/client.rb

@@ -0,0 +1,85 @@
+# frozen_string_literal: true
+
+require 'google/cloud/storage'
+require 'storazzo/common'
+
+module Storazzo
+  module GCS
+    class Client
+      include Storazzo::Common
+
+      attr_reader :storage, :project_id, :bucket_name
+
+      def initialize(project_id = nil, bucket_name = nil)
+        @project_id = project_id || autodetect_project_id
+        @bucket_name = bucket_name || ENV['GCS_BUCKET'] || "#{@project_id}-storazzo"
+        deb "GCS Client initialized for project: #{@project_id}, bucket: #{@bucket_name}"
+      end
+
+      def storage
+        @storage ||= Google::Cloud::Storage.new(project_id: @project_id)
+      end
+
+      def ensure_bucket_exists
+        return true if bucket_exists?(@bucket_name)
+
+        deb "Bucket #{@bucket_name} not found. Creating it in project #{@project_id}..."
+        storage.create_bucket(@bucket_name)
+        true
+      rescue StandardError => e
+        err "Failed to ensure/create bucket #{@bucket_name}: #{e.message}"
+        false
+      end
+
+      def autodetect_project_id
+        # Priority 1: Environment Variable
+        return ENV['GOOGLE_CLOUD_PROJECT'] if ENV['GOOGLE_CLOUD_PROJECT']
+        
+        # Priority 2: Config file
+        config = Storazzo::RicDiskConfig.instance
+        config.load
+        return config.project_id if config.project_id && config.project_id != 'YOUR-PROJECT-ID'
+
+        # Priority 3: GCloud CLI default
+        begin
+          `gcloud config get-value project 2>/dev/null`.strip
+        rescue StandardError
+          nil
+        end
+      end
+
+      def bucket_exists?(bucket_name)
+        bucket_name = bucket_name.gsub('gs://', '').split('/').first
+        !storage.bucket(bucket_name).nil?
+      rescue StandardError => e
+        warn "Error checking bucket #{bucket_name}: #{e.message}"
+        false
+      end
+
+      def list_buckets
+        storage.buckets.map(&:name)
+      end
+
+      def upload_file(local_path, bucket_name, remote_path)
+        bucket_name = bucket_name.gsub('gs://', '').split('/').first
+        bucket = storage.bucket(bucket_name)
+        raise "Bucket #{bucket_name} not found!" unless bucket
+
+        deb "Uploading #{local_path} to gs://#{bucket_name}/#{remote_path}..."
+        bucket.create_file(local_path, remote_path)
+      end
+
+      def download_file(bucket_name, remote_path, local_path)
+        bucket_name = bucket_name.gsub('gs://', '').split('/').first
+        bucket = storage.bucket(bucket_name)
+        raise "Bucket #{bucket_name} not found!" unless bucket
+
+        file = bucket.file(remote_path)
+        raise "Remote file #{remote_path} not found in bucket #{bucket_name}!" unless file
+
+        deb "Downloading gs://#{bucket_name}/#{remote_path} to #{local_path}..."
+        file.download(local_path)
+      end
+    end
+  end
+end

data/lib/storazzo/main.rb

@@ -15,11 +15,9 @@ module Storazzo
   # Arguments:
   #   message: (String) - optional
 
-  # Can be same name as Module: https://stackoverflow.com/questions/13261474/ruby-modules-and-classes-same-name-in-structure
-  module Storazzo
-    class Main
+  class Main
       require 'storazzo/colors'
-      extend Storazzo::Colors
+      extend ::Storazzo::Colors
 
       # version 1.2
       def self.say_hi(message = nil)
@@ -78,6 +76,5 @@ module Storazzo
         pwhite 'All tests END'
         # puts "All tests END"
       end
-    end
   end
 end