longleaf 0.1.0 → 0.2.0.pre.1

data/README.md

@@ -1,8 +1,11 @@
 # Longleaf
+Longleaf is a command-line tool which allows users to configure a set of storage locations and define custom sets of preservation services to run on their contents. These services are executed in response to applicable preservation events issued by clients. Its primary goal is to provide tools to create a simple and customizable preservation environment.  Longleaf:
 
-Welcome to your new gem! In this directory, you'll find the files you need to be able to package up your Ruby library into a gem. Put your Ruby code in the file `lib/longleaf`. To experiment with that code, run `bin/console` for an interactive prompt.
-
-TODO: Delete this and the text above, and describe your gem
+* Offers a predictable command-line interface and integrates with standard command-line tools.
+* Offers configurable and customizable criteria based preservation workflows.
+* Provides a base set of tools and a framework for building extensions.
+* Provides activity logging and notifications.
+* Performs preservation services only when required.
 
 ## Installation
 
@@ -22,22 +25,41 @@ Or install it yourself as:
 
 ## Usage
 
+#### Register a file
+In order to register a new file with Longleaf, use the register command:
+`longleaf register -c <config.yml> -f <path to file>`
+
+In the case that a file's content is replaced, the file can be re-registered by providing the `--force` flag.
+
 #### Validate configuration files
 Application configuration files can be validated prior to usage with the following command:
-`longleaf validate_config <config.yml>`
+`longleaf validate_config -c <config.yml>`
+
+#### Output and logging
+
+The primary output from Longleaf is directed to STDOUT, and contains both success and failure messages. If you would like to only return failure messages, you may provide the `--failure_only` flag.
+
+Additional logging is sent to STDERR. To control the level of logging, you may provide the `--log-level` parameter, which expects the standard [Ruby Logger levels](https://ruby-doc.org/stdlib-2.4.0/libdoc/logger/rdoc/Logger.html). The default log level is 'WARN'.
+
+Messages sent to STDOUT are duplicated to STDERR at 'INFO' level, so they are excluded by default. In order to store an ongoing log of activity and errors, you would perform the following:
+`longleaf <command> --log-level 'INFO' 2> /logs/longleaf.log`
 
 ## Development
 
 After checking out the repo, run `bin/setup` to install dependencies. Then, run `bundle exec rspec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
 
-To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+To run Longleaf with local changes without needing to do a local install, you may run `bundle exec exe/longleaf <command>`.
+
+To install this gem onto your local machine, run `bundle exec rake install`. This will allow you to run `longleaf <command>` and places the gem into `pkg/`. Note: Only files committed to git will be included in the installed gem.
+
+To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
 
 ## Contributing
 
-Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/longleaf.
+Bug reports and pull requests are welcome on GitHub at https://gitlab.lib.unc.edu/cdr/longleaf.
 
 
 ## License
 
-The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).
+The gem is available as open source under the terms of the [Apache License 2.0](http://www.apache.org/licenses/LICENSE-2.0).
 

data/lib/longleaf/candidates/file_selector.rb

@@ -0,0 +1,107 @@
+require 'longleaf/logging'
+
+module Longleaf
+  # Selects and allows for iteration over files which match a provided set of selection criteria
+  class FileSelector
+    include Longleaf::Logging
+    
+    # May only provide either file_paths or storage_locations
+    def initialize(file_paths: nil, storage_locations: nil, app_config:)
+      if nil_or_empty?(file_paths) && nil_or_empty?(storage_locations)
+        raise ArgumentError.new("Must provide either file paths or storage locations")
+      end
+      if !nil_or_empty?(file_paths) && !nil_or_empty?(storage_locations)
+        raise ArgumentError.new("Cannot provide both file paths and storage locations")
+      end
+      @app_config = app_config
+      # The top level paths targeted by this selector
+      @target_paths = file_paths
+      # The set of storage locations to select file paths from
+      @storage_locations = storage_locations
+      # Validate that the selected storage locations are known
+      unless @storage_locations.nil?
+        locations = @app_config.location_manager.locations
+        @storage_locations.each do |loc_name|
+          unless locations.key?(loc_name)
+            raise StorageLocationUnavailableError.new("Cannot select unknown storage location #{loc_name}.")
+          end
+        end
+      end
+    end
+    
+    # @return [Array] a list of top level paths from which files will be selected
+    def target_paths
+      # If starting from locations, initialize by expanding locations out to their actual paths
+      if @target_paths.nil? && !@storage_locations.nil?
+        @target_paths = Array.new
+        @storage_locations.each do |loc_name|
+          @target_paths << @app_config.location_manager.locations[loc_name].path
+        end
+      end
+      
+      @target_paths
+    end
+    
+    # Get the next file path for this selector.
+    # @return [String] an absolute path to the next file targeted by this selector,
+    # or nil if no more files selected
+    def next_path
+      if @paths.nil?
+        # Start the paths listing out from the targetted set of paths for this selector
+        # In reverse order since using a LIFO structure
+        @paths = target_paths.reverse
+      end
+
+      # No more paths to return
+      return nil if @paths&.empty?
+      
+      # Get the most recently added path for depth first traversal of selected paths
+      path = @paths.pop
+      until path.nil? do
+        @app_config.location_manager.verify_path_in_location(path)
+        
+        if File.exist?(path)
+          if File.directory?(path)
+            logger.debug("Expanding directory #{path}")
+            # For a directory, add all children to file_paths
+            Dir.entries(path).sort.reverse.each do |child|
+              @paths << File.join(path, child) unless child == '.' or child == '..'
+            end
+          else
+            logger.debug("Returning file #{path}")
+            return path
+          end
+        else
+          raise InvalidStoragePathError.new("File #{path} does not exist.")
+        end
+        
+        # Returned path was not a suitable file, try the next path
+        path = @paths.pop
+      end
+    end
+    
+    # return [Array] a list of all storage locations being targeted by this selector
+    def storage_locations
+      # Determine what storage_locations are represented by the given file paths
+      if @storage_locations.nil? && !@target_paths.nil?
+        loc_set = Set.new
+        @target_paths.each do |path|
+          loc = @app_config.location_manager.get_location_by_path(path)
+          loc_set.add(loc.name) unless loc.nil?
+        end
+        @storage_locations = loc_set.to_a
+      end
+      
+      if @storage_locations.nil?
+        @storage_locations = Array.new
+      end
+      
+      @storage_locations
+    end
+    
+    private
+    def nil_or_empty?(value)
+      value.nil? || value.empty?
+    end
+  end
+end

data/lib/longleaf/candidates/service_candidate_filesystem_iterator.rb

@@ -0,0 +1,99 @@
+require 'longleaf/services/service_manager'
+require 'longleaf/services/metadata_deserializer'
+require 'longleaf/events/event_names'
+require 'longleaf/errors'
+require 'longleaf/logging'
+require 'time'
+
+module Longleaf
+  # Iterator for getting file candidates which have services which need to be run.
+  # Implementation uses metadata files directly from the filesystem for determinations
+  # about service status.
+  class ServiceCandidateFilesystemIterator
+    include Longleaf::Logging
+    
+    def initialize(file_selector, event, app_config, force = false)
+      @file_selector = file_selector
+      @event = event
+      @app_config = app_config
+      @force = force
+    end
+    
+    # Get the file record for the next candidate which needs services run which match the 
+    # provided file_selector
+    # @return [FileRecord] file record of the next candidate with services needing to be run,
+    # or nil if there are no more candidates.
+    def next_candidate
+      loop do
+        begin
+          next_path = @file_selector.next_path
+          return nil if next_path.nil?
+          
+          logger.debug("Evaluating candidate #{next_path}")
+          storage_loc = @app_config.location_manager.get_location_by_path(next_path)
+          file_rec = FileRecord.new(next_path, storage_loc)
+      
+          # Skip over unregistered files
+          if !file_rec.metadata_present?
+            logger.debug("Ignoring unregistered file #{next_path}")
+            next
+          end
+          
+          file_rec.metadata_record = MetadataDeserializer.deserialize(file_path: file_rec.metadata_path,
+              digest_algs: storage_loc.metadata_digests)
+      
+          # Return the file record if it needs any services run
+          return file_rec if needs_run?(file_rec)
+        rescue InvalidStoragePathError => e
+          logger.warn("Skipping candidate file: #{e.message}")
+        end
+      end
+    end
+    
+    # Iterate through the candidates in this object and execute the provided block with each
+    # candidate. A block is required.
+    def each
+      file_rec = next_candidate
+      until file_rec.nil?
+        yield file_rec
+        
+        file_rec = next_candidate
+      end
+    end
+    
+    private
+    # Returns true if the file record contains any services which need to be run
+    def needs_run?(file_rec)
+      md_rec = file_rec.metadata_record
+      storage_loc = file_rec.storage_location
+      service_manager = @app_config.service_manager
+      
+      # File is not a valid candidate for services if it is deregistered, unless performing cleanup
+      if @event != EventNames::CLEANUP && md_rec.deregistered?
+        logger.debug("Skipping deregistered file: #{file_rec.path}")
+        return false
+      end
+      
+      expected_services = service_manager.list_services(
+          location: storage_loc.name,
+          event: @event)
+      
+      # When in force mode, candidate needs a run as long as there are any services configured for it.
+      if @force && expected_services.length > 0
+        logger.debug("Forcing run needed for file: #{file_rec.path}")
+        return true
+      end
+          
+      expected_services.each do |service_name|
+        if service_manager.service_needed?(service_name, md_rec)
+          logger.debug("Service #{service_name} needed for file: #{file_rec.path}")
+          return true
+        end
+      end
+      
+      logger.debug("Run not needed for file: #{file_rec.path}")
+      # No services needed to be run for this file
+      false
+    end
+  end
+end

data/lib/longleaf/candidates/service_candidate_locator.rb

@@ -0,0 +1,18 @@
+require 'longleaf/candidates/service_candidate_filesystem_iterator'
+
+module Longleaf
+  # Service which locates files that have services which need to be performed on them.
+  class ServiceCandidateLocator
+    def initialize(app_config)
+      @app_config = app_config
+    end
+    
+    # Get a iterator of the candidates matching the given FileSelector which need services run.
+    # @param file_selector [FileSelector] selector identifying the files to pull candidates from.
+    # @return an iterator
+    def candidate_iterator(file_selector, event, force = false)
+      # Get filesystem based implementation
+      ServiceCandidateFilesystemIterator.new(file_selector, event, @app_config, force)
+    end
+  end
+end

data/lib/longleaf/cli.rb

@@ -2,16 +2,24 @@ require 'thor'
 require 'yaml'
 require 'longleaf/logging'
 require 'longleaf/errors'
+require 'longleaf/commands/deregister_command'
 require 'longleaf/commands/validate_config_command'
+require 'longleaf/commands/validate_metadata_command'
 require 'longleaf/commands/register_command'
+require 'longleaf/commands/preserve_command'
+require 'longleaf/candidates/file_selector'
 
 module Longleaf
+  # Main commandline interface setup for Longleaf using Thor.
   class CLI < Thor
     include Longleaf::Logging
     
     class_option(:config, :aliases => "-c",
         :default => ENV['LONGLEAF_CFG'],
+        :required => true,
         :desc => 'Absolute path to the application configuration used for this command. By default, the value of the environment variable LONGLEAF_CFG is used.')
+    class_option(:load_path, :aliases => "-I",
+        :desc => 'Specify comma seperated directories to add to the $LOAD_PATH, which can be used to specify additional paths from which to load preservation services.')
     # Logging/output options
     class_option(:failure_only,
         :type => :boolean,
@@ -41,7 +49,8 @@ module Longleaf
       setup_logger(options)
       
       config_path = options[:config]
-      file_paths = options[:file]&.split(/\s*,\s*/)
+      app_config_manager = load_application_config(options[:config])
+      file_selector = create_file_selector(options[:file], nil, app_config_manager)
       if options[:checksums]
         checksums = options[:checksums]
         # validate checksum list format, must a comma delimited list of prefix:checksums
@@ -54,16 +63,75 @@ module Longleaf
         end
       end
       
-      command = Longleaf::RegisterCommand.new(config_path)
-      exit command.execute(file_paths: file_paths, force: options[:force], checksums: checksums)
+      command = RegisterCommand.new(app_config_manager)
+      exit command.execute(file_selector: file_selector, force: options[:force], checksums: checksums)
+    end
+    
+    desc "deregister", "Deregister files with Longleaf"
+    method_option(:file, :aliases => "-f", 
+        :required => true,
+        :desc => 'File or files to deregister. Paths must be absolute. If multiple files are provided, they must be comma separated.')
+    method_option(:force,
+        :type => :boolean, 
+        :default => false,
+        :desc => 'Force the deregistration of already deregistered files.')
+    # Deregister event command
+    def deregister
+      setup_logger(options)
+      
+      config_path = options[:config]
+      app_config_manager = load_application_config(options[:config])
+      file_selector = create_file_selector(options[:file], nil, app_config_manager)
+      
+      command = DeregisterCommand.new(app_config_manager)
+      exit command.execute(file_selector: file_selector, force: options[:force])
+    end
+    
+    desc "preserve", "Perform preservation services on files with Longleaf"
+    method_option(:file, :aliases => "-f", 
+        :required => false,
+        :desc => 'File or files to preserve. Paths must be absolute. If multiple files are provided, they must be comma separated.')
+    method_option(:location, :aliases => "-s",
+        :required => false,
+        :desc => 'Name or comma separated names of storage locations to preserve.')
+    method_option(:force,
+        :type => :boolean, 
+        :default => false,
+        :desc => 'Force the execution of preservation services, disregarding scheduling information.')
+    def preserve
+      setup_logger(options)
+      
+      extend_load_path(options[:load_path])
+      app_config_manager = load_application_config(options[:config])
+      file_selector = create_file_selector(options[:file], options[:location], app_config_manager)
+      
+      command = PreserveCommand.new(app_config_manager)
+      exit command.execute(file_selector: file_selector, force: options[:force])
     end
     
-    desc "validate_config [CONFIG_PATH]", "Validate an application configuration file"
+    desc "validate_config", "Validate an application configuration file, provided using --config."
     # Application configuration validation command
-    def validate_config(config_path)
+    def validate_config
+      setup_logger(options)
+      
+      exit Longleaf::ValidateConfigCommand.new(options[:config]).execute
+    end
+    
+    desc "validate_metadata", "Validate metadata files."
+    method_option(:file, :aliases => "-f", 
+        :required => false,
+        :desc => 'File or files of which to validate the metadata. Paths must be absolute. If multiple files are provided, they must be comma separated.')
+    method_option(:location, :aliases => "-s",
+        :required => false,
+        :desc => 'Name or comma separated names of storage locations to validate.')
+    # File metadata validation command
+    def validate_metadata
       setup_logger(options)
       
-      exit Longleaf::ValidateConfigCommand.new(config_path).execute
+      app_config_manager = load_application_config(options[:config])
+      file_selector = create_file_selector(options[:file], options[:location], app_config_manager)
+      
+      exit Longleaf::ValidateMetadataCommand.new(app_config_manager).execute(file_selector: file_selector)
     end
     
     no_commands do
@@ -73,6 +141,34 @@ module Longleaf
             options[:log_format],
             options[:log_datetime])
       end
+      
+      def load_application_config(config_path)
+        begin
+          app_manager = ApplicationConfigDeserializer.deserialize(config_path)
+        rescue ConfigurationError => err
+          logger.failure("Failed to load application configuration due to the following issue:\n#{err.message}")
+          exit 1
+        end
+      end
+      
+      def create_file_selector(file_paths, storage_locations, app_config_manager)
+        file_paths = file_paths&.split(/\s*,\s*/)
+        storage_locations = storage_locations&.split(/\s*,\s*/)
+        
+        begin
+          FileSelector.new(file_paths: file_paths, storage_locations: storage_locations, app_config: app_config_manager)
+        rescue ArgumentError => e
+          logger.failure(e.message)
+          exit 1
+        end
+      end
+      
+      def extend_load_path(load_paths)
+        load_paths = load_paths&.split(/\s*,\s*/)
+        if !load_paths.nil?
+          load_paths.each { |path| $LOAD_PATH.unshift(path) }
+        end
+      end
     end
   end
 end

data/lib/longleaf/commands/deregister_command.rb

@@ -0,0 +1,50 @@
+require 'longleaf/services/application_config_deserializer'
+require 'longleaf/events/deregister_event'
+require 'longleaf/models/file_record'
+require 'longleaf/events/event_names'
+require 'longleaf/events/event_status_tracking'
+require 'longleaf/services/metadata_deserializer'
+
+module Longleaf
+  # Command for deregistering files with longleaf
+  class DeregisterCommand
+    include Longleaf::EventStatusTracking
+    
+    def initialize(app_manager)
+      @app_manager = app_manager
+    end
+
+    # Execute the deregister command on the given parameters
+    # @param file_selector [FileSelector] selector for files to deregister
+    # @param force [Boolean] force flag
+    # @return [Integer] status code
+    def execute(file_selector:, force: false)
+      begin
+        # Perform deregister events on each of the file paths provided
+        loop do
+          f_path = file_selector.next_path
+          break if f_path.nil?
+          
+          storage_location = @app_manager.location_manager.get_location_by_path(f_path)
+        
+          file_rec = FileRecord.new(f_path, storage_location)
+          unless file_rec.metadata_present?
+            raise DeregistrationError.new("Cannot deregister #{f_path}, file is not registered.")
+          end
+          
+          file_rec.metadata_record = MetadataDeserializer.deserialize(file_path: file_rec.metadata_path,
+              digest_algs: storage_location.metadata_digests)
+          
+          event = DeregisterEvent.new(file_rec: file_rec, force: force, app_manager: @app_manager)
+          track_status(event.perform)
+        end
+      rescue DeregistrationError, InvalidStoragePathError, StorageLocationUnavailableError => err
+        record_failure(EventNames::DEREGISTER, nil, err.message)
+      rescue => err
+        record_failure(EventNames::DEREGISTER, error: err)
+      end
+      
+      return_status
+    end
+  end
+end