derivative-rodeo 0.2.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 20e512d7162170875d60f90ff48bba694237ac7d31f38d806d2b87f570536c1c
+  data.tar.gz: a1311ea39a3994b4d24ffdbdbade62b7fbb15d2c326a3b510607f315fb4dd865
+SHA512:
+  metadata.gz: 157a9e276c6cefe739137fbe17e783557d0317dcee531cd353ad86987bda33ad55c2ada8179e254b306a467d01f8f759a6e89fc91b8cbcf6e968cf5a28a9037b
+  data.tar.gz: 5bd45db467194cf1e8af7f7e1ed625c2b3898d011f20a581a9a55a2ccbb7be56ca8c276752b7bef41e2c1d5efa4737d1291f83211dd69cd18e4f7caeed25fef2

data/Gemfile

@@ -0,0 +1,6 @@
+# frozen_string_literal: true
+
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in derivative_rodeo.gemspec
+gemspec

data/LICENSE

@@ -0,0 +1,15 @@
+Copyright 2023 Software Services by Scientist.com
+
+Additional copyright may be held by others, as reflected in the commit history.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

data/README.md

@@ -0,0 +1,251 @@
+<!-- START doctoc generated TOC please keep comment here to allow auto update -->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+**Table of Contents**  *generated with [DocToc](https://github.com/thlorenz/doctoc)*
+
+- [DerivativeRodeo](#derivativerodeo)
+  - [Process Life Cycle](#process-life-cycle)
+  - [Concepts](#concepts)
+    - [Common Storage](#common-storage)
+    - [Related Files](#related-files)
+    - [Sequence Diagram](#sequence-diagram)
+  - [Installation](#installation)
+  - [Usage](#usage)
+  - [Technical Overview of the DerivativeRodeo](#technical-overview-of-the-derivativerodeo)
+    - [Generators](#generators)
+      - [Interface(s)](#interfaces)
+      - [Supported Generators](#supported-generators)
+      - [Registered Generators](#registered-generators)
+    - [Storage Locations](#storage-locations)
+      - [Supported Storage Locations](#supported-storage-locations)
+  - [Development](#development)
+    - [Logging in Test Environment](#logging-in-test-environment)
+  - [Contributing](#contributing)
+
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
+
+# DerivativeRodeo
+
+“This ain’t my first rodeo.” (an idiomatic American slang for “I’m prepared for what comes next.”)
+
+The `DerivativeRodeo` "moves" files from one storage location (e.g. *input*) to one or more storage locations (e.g. *output*) via a generator.
+
+- [Storage Location](./lib/derivative_rodeo/storage_locations/base_location.rb) :: where we can expect to find a file.
+- [Generator](./lib/derivative_rodeo/generators/base_generator.rb) :: a process to transform a file into another file.
+
+## Process Life Cycle
+
+In the case of a *input* storage location, we expect that the underlying file pointed at by the *input* storage location exists.  After all we can't move what we don't have.
+
+In the case of a *output* storage location, we expect that the underlying file will exist after the generator has completed.  The *output* storage location *could* already exist or we might need to generate the file for the *output* location.
+
+During the generator's process, we need to have a working copy of both the *input* and *output* file.  This is done by creating a temporary file.
+
+In the case of the *input*, the creation of that temporary file involves getting the file from the *input* storage location.  In the case of the *output*, we create a temporary file that the *output* storage location then knows how to move to the resulting place.
+
+![Storage Lifecycle](./artifacts/derivative_rodeo-generator_storage_lifecycle.png)
+
+The above Storage Lifecycle diagram is as follows: `input location` to `input tmp file` to `generator` to `output tmp file` to `output location`.
+
+*Note:* We've designed and implemented the data life cycle to automatically clean-up the temporary files as the generator completes.  In this way we can use the smallest working space possible.  A design decision that helps run `DerivativeRodeo` within distributed clusters (e.g. AWS Serverless).
+
+## Concepts
+
+![Overview](./artifacts/derivative_rodeo-overview.png)
+
+<details>
+<summary>The PlantUML Text for the Overview Diagram</summary>
+
+```plantuml
+@startuml
+!theme amiga
+
+cloud "Source 1" as S1
+cloud "Source 2" as S2
+cloud "Source 3" as S3
+
+storage "IMAGEs" as IMAGEs
+storage "HOCRs" as HOCRs
+storage "TXTs" as TXTs
+
+control Preprocess as G1
+
+S1 -down-> G1
+S2 -down-> G1
+S3 -down-> G1
+
+G1 -down-> IMAGEs
+G1 -down-> HOCRs
+G1 -down-> TXTs
+
+control Import as I1
+
+IMAGEs -down-> I1
+HOCRs -down-> I1
+TXTs -down-> I1
+
+package FileSet as FileSet1 {
+	file Image1
+	file Hocr1
+	file Txt1
+}
+package FileSet as FileSet2 {
+	file Image2
+	file Hocr2
+	file Txt2
+}
+
+I1 -down-> FileSet1
+I1 -down-> FileSet2
+
+@enduml
+
+```
+
+</details>
+
+### Common Storage
+
+In this case, <dfn>common storage</dfn> could mean the storage where we're writing all pre-processing of files.  Or it could mean the storage where we're writing for application access (e.g. [Fedora Commons](https://fedora.lyrasis.org) for a [Hyrax](https://github.com/samvera/hyrax) application).
+
+In other words, the `DerivativeRodeo` is part of moving files from one location to another, and ensuring that at each step we have all of the expected files we want.
+
+### Related Files
+
+This is not strictly related to <dfn>Hyrax's FileSet</dfn>, that is a set of files in which one is considered the original and all others are _derivatives_ of the original.
+
+However it is helpful to think in those terms; files that have a significant relation to each other; one derived from the other.  For example an original PDF and it's extracted text would be two significantly related files.
+
+### Sequence Diagram
+
+![Sequence Diagram](./artifacts/derivative_rodeo-sequence-diagram.png)
+
+<details>
+<summary>The PlantUML Text for the Sequence Diagram</summary>
+
+```plantuml
+@startuml
+!theme amiga
+
+actor Instigator
+database S3
+control AWS
+queue SQS
+control SpaceStone
+control DerivativeRodeo
+collections From
+collections To
+Instigator -> S3 : "Upload bucket\nof files associated\n with FileSet"
+S3 -> AWS : "AWS enqueues\nthe bucket"
+AWS -> SQS : "AWS adds to SQS"
+SQS -> SpaceStone : "SQS invokes\nSpaceStone method"
+SpaceStone -> DerivativeRodeo : "SpaceStone calls\n DerivativeRodeo"
+DerivativeRodeo --> S3 : "Request file for\ntemporary processing"
+S3 --> From : "Write requested\n file to\ntemporary storage"
+DerivativeRodeo <-- From
+DerivativeRodeo -> To : "Generate derivative\n writing to local\n processing storage."
+To --> S3 : "Write file\n to S3 Bucket"
+DerivativeRodeo <-- To : "Return to DerivativeRodeo\n with generated URIs"
+SpaceStone <- DerivativeRodeo : "Return generated\n URIs"
+SpaceStone -> SQS : "Optionally enqueue\nfurther work"
+@enduml
+```
+</details>
+
+Given a single original file in a previous home, we are copying that original file (and derivatives) to various locations:
+
+- From previous home to S3.
+- From S3 to local temporary storage (for processing).
+- Create a derivative temporary file based on existing file.
+- Copying derivative temporary file to S3.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'derivative-rodeo'
+```
+
+(Due to historical reasons the gem name is `derivative-rodeo` even though the repository is `derivative_rodeo`.  The following "require" methods will work:
+
+- `require 'derivative_rodeo'`
+- `require 'derivative-rodeo'`
+- `require 'derivative/rodeo'`
+
+And then execute: `$ bundle install`
+
+Be aware that you need `pdfinfo` command line tool installed for this gem to run specs or when using PDF functionality.
+
+## Usage
+
+TODO
+
+## Technical Overview of the DerivativeRodeo
+
+### Generators
+
+Generators are responsible for ensuring that we have the file associated with the generator.  For example, the [HocrGenerator](./lib/derivative_rodeo/generators/hocr_generator.rb) is responsible for ensuring that we have the `.hocr` file in the expected desired storage location.
+
+#### Interface(s)
+
+Generators must have an initializer and build command:
+
+- `.new(array_of_file_urls, output_location_template, preprocessed_location_template)`
+- `#generated_files` (executes the generators actions) and returns array of files
+- `#generated_uris` (executes the generators actions) and returns array of output uris
+
+#### Supported Generators
+
+Below is the current list of generators.
+
+- [HocrGenerator](./lib/derivative_rodeo/generators/hocr_generator.rb) :: generated tesseract files from images, also creates monocrhome files as a prestep
+- [MonochromeGenerator](./lib/derivative_rodeo/generators/monochrome_generator.rb) :: converts images to monochrome
+- [CopyGenerator](./lib/derivative_rodeo/generators/copy_generator.rb) :: sends a set of uris to another location. For example from <abbr title="Simple Storage Service">S3</abbr> to <abbr title="Simple Queue Service">SQS</abbr> or from filesystem to S3.
+- [PdfSplitGenerator](./lib/derivative_rodeo/generators/pdf_split_generator.rb) :: split a PDF into one image per page
+- [WordCoordinatesGenerator](./lib/derivative_rodeo/generators/word_coordinates_generator.rb) :: create a JSON file representing the words and coordinates (derived from the `.hocr` file).
+
+#### Registered Generators
+
+TODO: We want to expose a list of registered generators
+
+### Storage Locations
+
+Storage locations are where we put things.  Each location has a specific implementation but is expected to inherit from the  [DerivativeRodeo::StorageLocation::BaseLocation](./lib/derivative_rodeo/storage_adapters/base_adapter.rb).
+
+`DerivativeRodeo::StorageLocation::BaseLocation.locations` method tracks the registered locations.
+
+The location represents where the file *should* be.
+
+#### Supported Storage Locations
+
+Storage locations follow a [URI pattern](https://en.wikipedia.org/wiki/Uniform_Resource_Identifier#Example_URIs)
+
+- `file://` :: “local” file system storage
+- `s3://` :: <abbr title="Amazon Web Service">AWS</abbr>’s <abbr title="Simple Storage Service">S3</abbr> storage system
+- `sqs://` :: <abbr title="Amazon Web Service">AWS</abbr>’s <abbr title="Simple Queue Service">SQS</abbr>
+
+## Development
+
+- Checkout the repository: `git clone https://github.com/scientist-softserv/derivative_rodeo`
+- Install dependencies: `cd derivative_rodeo; bundle install`
+- Install git hooks: `rake install_hooks`
+- Install binaries:
+  - `pdfinfo`: provided by poppler (e.g. `brew install poppler`)
+  - GhostScript (e.g. `gs`): run `brew install gs`
+
+Then go about writing your code and documentation.
+
+The git hooks call `rake default` which will:
+
+- Amend the table of contents of this file
+- Run `rubocop`
+- Validate yard documentation (see http://rubydoc.info/gems/yard/file/docs/Tags.md#List_of_Available_Tags for help correcting warnings)
+- Run `rspec` with `simplecov`
+
+### Logging in Test Environment
+
+Throughout the `DerivativeRodeo` we log some activity.  In the typical test run, the logs are overly chatty.  If you want the more chatty logs run the following: `DEBUG=t rspec`.
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/scientist-softserv/derivative_rodeo.

data/Rakefile

@@ -0,0 +1,42 @@
+# frozen_string_literal: true
+
+require 'bundler/gem_tasks'
+require 'rspec/core/rake_task'
+require 'rubocop/rake_task'
+
+desc 'Run style checker'
+RuboCop::RakeTask.new(:rubocop) do |task|
+  task.fail_on_error = true
+end
+
+desc 'Install commit hooks to ensure better practices'
+task :install_hooks do
+  require 'fileutils'
+  Dir.glob('./git-hooks/*').each do |hook|
+    next if File.file?("./.git/hooks/#{File.basename(hook)}")
+
+    puts "Installing #{File.basename(hook)} git hook"
+    FileUtils.cp(hook, './.git/hooks/')
+  end
+end
+
+require 'yard'
+YARD::Rake::YardocTask.new do |t|
+  t.options = ['--fail-on-warning']
+end
+
+RSpec::Core::RakeTask.new(:spec)
+
+desc 'Generate table of contents for README.md'
+task :doctoc do
+  if `which doctoc`.strip.empty?
+    $stdout.puts 'Skipping doctoc generation; install via "npm install -g doctoc"'
+  else
+    $stdout.puts 'Generating table of contents for README.md'
+    `doctoc README.md`
+  end
+end
+
+task ci: %i[doctoc rubocop yard spec]
+
+task default: %i[ci]

data/derivative_rodeo.gemspec

@@ -0,0 +1,54 @@
+# frozen_string_literal: true
+
+require_relative 'lib/derivative_rodeo/version'
+
+Gem::Specification.new do |spec|
+  # Renaming to reflect that we previously registered 'derivative-rodeo' and Rubygems guards against
+  # names that are close in resemblence.
+  spec.name          = 'derivative-rodeo'
+  spec.version       = DerivativeRodeo::VERSION
+  spec.authors       = ['Rob Kaufman', 'Jeremy Friesen']
+  spec.email         = ['rob@notch8.com', 'jeremy.n.friesen@gmail.com']
+
+  spec.summary = 'An ETL Ecosystem for Derivative Processing.'
+  spec.description = spec.summary
+  spec.homepage = 'https://github.com/scientist-softserv/derivative_rodeo'
+  spec.required_ruby_version = '>= 2.7.0'
+  spec.licenses = ['APACHE-2.0']
+
+  spec.metadata['homepage_uri'] = spec.homepage
+  spec.metadata['source_code_uri'] = spec.homepage
+
+  # Specify which files should be added to the gem when it is released.
+  # The `git ls-files -z` loads the files in the RubyGem that have been added into git.
+  # spec.files = Dir.chdir(__dir__) do
+  #   `git ls-files -z`.split("\x0").reject do |f|
+  #     (f == __FILE__) || f.match(%r{\A(?:(?:bin|test|spec|features)/|\.(?:git|travis|circleci)|appveyor)})
+  #   end
+  # end
+  spec.files = Dir['lib/**/*'].keep_if { |file| File.file?(file) } + %w[Gemfile LICENSE README.md Rakefile derivative_rodeo.gemspec]
+  spec.bindir = 'exe'
+  spec.executables = spec.files.grep(%r{\Aexe/}) { |f| File.basename(f) }
+  spec.require_paths = ['lib']
+
+  spec.add_dependency 'activesupport', '>= 5'
+  spec.add_dependency 'aws-sdk-s3'
+  spec.add_dependency 'aws-sdk-sqs'
+  spec.add_dependency 'httparty'
+  spec.add_dependency 'marcel'
+  spec.add_dependency 'mime-types'
+  spec.add_dependency 'mini_magick'
+  spec.add_dependency 'nokogiri'
+
+  spec.add_development_dependency 'bixby'
+  spec.add_development_dependency 'byebug'
+  # spec.add_development_dependency 'hydra-file_characterization'
+  spec.add_development_dependency 'rspec', '~> 3.0'
+  spec.add_development_dependency 'rake', '~> 13.0'
+  spec.add_development_dependency 'simplecov'
+  spec.add_development_dependency 'yard-activerecord'
+  spec.add_development_dependency 'rspec-its'
+  spec.add_development_dependency 'shoulda-matchers'
+  spec.add_development_dependency 'solargraph'
+  spec.add_development_dependency 'yard'
+end

data/lib/derivative/rodeo.rb

@@ -0,0 +1,3 @@
+# frozen_string_literal: true
+
+require_relative "../derivative_rodeo"

data/lib/derivative-rodeo.rb

@@ -0,0 +1,3 @@
+# frozen_string_literal: true
+
+require_relative './derivative_rodeo'

data/lib/derivative_rodeo/configuration.rb

@@ -0,0 +1,95 @@
+# frozen_string_literal: true
+
+require 'mime/types'
+require 'logger'
+module DerivativeRodeo
+  ##
+  # @api public
+  #
+  # This class is responsible for the consistent configuration of the "application" that leverages
+  # the {DerivativeRodeo}.
+  #
+  # This configuration helps set defaults for storage locations and generators.
+  class Configuration
+    ##
+    # Allows AWS configuration to be set via environment variables by declairing them in the configuration
+    # class as follows:
+    #
+    # @example
+    #
+    #  aws_config prefix: 's3', name: 'region', default: 'us-east-1'
+    #
+    # @param prefix [String]
+    # @param name [String]
+    # @param default [String] (optional)
+    def self.aws_config(prefix:, name:, default: nil)
+      aws_config_getter(prefix: prefix, name: name, default: default)
+      aws_config_setter(prefix: prefix, name: name)
+    end
+
+    def self.aws_config_getter(prefix:, name:, default: nil)
+      define_method "aws_#{prefix}_#{name}" do
+        val = instance_variable_get("@aws_#{prefix}_#{name}")
+        return val if val
+
+        val = ENV["AWS_#{prefix.upcase}_#{name.upcase}"] ||
+              ENV["AWS__#{name.upcase}"] ||
+              ENV["AWS_DEFAULT_#{name.upcase}"] ||
+              default
+        instance_variable_set("@aws_#{prefix}_#{name}", val)
+      end
+    end
+    private_class_method :aws_config_getter
+
+    def self.aws_config_setter(prefix:, name:)
+      define_method "aws_#{prefix}_#{name}=" do |val|
+        instance_variable_set("@aws_#{prefix}_#{name}", val)
+      end
+    end
+    private_class_method :aws_config_setter
+
+    def initialize
+      @logger = if defined?(Rails)
+                  Rails.logger
+                else
+                  # By default, minimize the chatter of the specs.  Add ENV['DEBUG'] to expose the
+                  # chatter.
+                  Logger.new($stderr, level: Logger::FATAL)
+                end
+      yield self if block_given?
+    end
+
+    ##
+    # @return [Logger]
+    attr_accessor :logger
+
+    ##
+    # @!group AWS S3 Configuration
+    #
+    # Various AWS items for {StorageLocations::S3Location}. These can be set from the ENV or the configuration block
+    #
+    # @note
+    #
+    #   The order we use is:
+    #   * `config.aws_s3_<variable_name> = value`
+    #   * `AWS_S3_<variable_name>`
+    #   * `AWS_<variable_name>`
+    #   * `AWS_DEFAULT_<variable_name>`
+    #   * default
+    #
+    # @return [String]
+
+    aws_config prefix: 's3', name: 'region', default: 'us-east-1'
+    aws_config prefix: 's3', name: 'bucket'
+    aws_config prefix: 's3', name: 'access_key_id'
+    aws_config prefix: 's3', name: 'secret_access_key'
+
+    aws_config prefix: 'sqs', name: 'region', default: 'us-east-1'
+    aws_config prefix: 'sqs', name: 'queue'
+    aws_config prefix: 'sqs', name: 'account_id'
+    aws_config prefix: 'sqs', name: 'access_key_id'
+    aws_config prefix: 'sqs', name: 'secret_access_key'
+    aws_config prefix: 'sqs', name: 'max_batch_size', default: 10
+    # @!endgroup AWS SQS Configurations
+  end
+end

data/lib/derivative_rodeo/errors.rb

@@ -0,0 +1,56 @@
+# frozen_string_literal: true
+module DerivativeRodeo
+  ##
+  # A module namespace for establishing the possible errors that the {DerivativeRodeo} could raise.
+  # The rodeo could raise other errors, but these are the ones we've named.
+  module Errors
+    ##
+    # That which all DerivativeRodeo errors shall extend!
+    class Error < StandardError; end
+
+    ##
+    # Raised when a file uri is passed in that does not contain a storage adapter part before the ://
+    class StorageLocationMissing < Error
+      def initialize(file_uri: '')
+        super("#{file_uri} does not contain an adapter. Should look like file:///my_dir/myfile or s3://bucket_name/location/file_name. The part before the :// is used to select the storage adapter.") # rubocop:disable Layout/LineLength
+      end
+    end
+
+    ##
+    # Raised when a storage adapter is called for but does not exist in the registered adapter list
+    class StorageLocationNotFoundError < Error
+      def initialize(location_name: '')
+        super("Could not find the adapter #{location_name}. Make sure it is required and registerd properly.")
+      end
+    end
+
+    ##
+    # Raised when a storage adapter is called for but does not exist in the registered adapter list
+    class MaxQueueSize < Error
+      def initialize(batch_size:)
+        super("Batch size #{batch_size} is larger than the max queue size #{DerivativeRodeo.config.aws_sqs_max_batch_size}")
+      end
+    end
+
+    ##
+    # Raised when AWS bucket does not exist or is not accessible by current permissions
+    class BucketMissingError < Error
+      def initialize
+        super("Bucket part missing #{file_uri}")
+      end
+    end
+
+    ##
+    # Raised when trying to write a tmp file that does not exist
+    class FileMissingError < Error
+    end
+
+    ##
+    # Raised because the Generator class must declare an extension for the output file extension
+    class ExtensionMissingError < Error
+      def initialize(klass: '')
+        super("Extension must be declared in the Generator class #{klass}")
+      end
+    end
+  end
+end