libis-rosetta_checker 1.0.0

data/.rspec

@@ -0,0 +1,3 @@
+--format documentation
+--color
+--require spec_helper

data/.travis.yml

@@ -0,0 +1,5 @@
+sudo: false
+language: ruby
+rvm:
+  - 2.4.0
+before_install: gem install bundler -v 1.16.0

data/CODE_OF_CONDUCT.md

@@ -0,0 +1,74 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as
+contributors and maintainers pledge to making participation in our project and
+our community a harassment-free experience for everyone, regardless of age, body
+size, disability, ethnicity, gender identity and expression, level of experience,
+nationality, personal appearance, race, religion, or sexual identity and
+orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment
+include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or
+advances
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic
+  address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable
+behavior and are expected to take appropriate and fair corrective action in
+response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies both within project spaces and in public spaces
+when an individual is representing the project or its community. Examples of
+representing a project or community include using an official project e-mail
+address, posting via an official social media account, or acting as an appointed
+representative at an online or offline event. Representation of a project may be
+further defined and clarified by project maintainers.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting the project team at kris.dekeyser@libis.be. All
+complaints will be reviewed and investigated and will result in a response that
+is deemed necessary and appropriate to the circumstances. The project team is
+obligated to maintain confidentiality with regard to the reporter of an incident.
+Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good
+faith may face temporary or permanent repercussions as determined by other
+members of the project's leadership.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
+available at [http://contributor-covenant.org/version/1/4][version]
+
+[homepage]: http://contributor-covenant.org
+[version]: http://contributor-covenant.org/version/1/4/

data/Gemfile

@@ -0,0 +1,6 @@
+source "https://rubygems.org"
+
+git_source(:github) {|repo_name| "https://github.com/#{repo_name}" }
+
+# Specify your gem's dependencies in libis-rosetta_checker.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2017 LIBIS (Kris Dekeyser)
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,43 @@
+# Libis::RosettaChecker
+
+Welcome to your new gem! In this directory, you'll find the files you need to be able to package up your Ruby library into a gem. Put your Ruby code in the file `lib/libis/rosetta_checker`. To experiment with that code, run `bin/console` for an interactive prompt.
+
+TODO: Delete this and the text above, and describe your gem
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'libis-rosetta_checker'
+```
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install libis-rosetta_checker
+
+## Usage
+
+TODO: Write usage instructions here
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/libis-rosetta_checker. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
+
+## Code of Conduct
+
+Everyone interacting in the Libis::RosettaChecker project’s codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/[USERNAME]/libis-rosetta_checker/blob/master/CODE_OF_CONDUCT.md).

data/Rakefile

@@ -0,0 +1,6 @@
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+
+task :default => :spec

data/bin/console

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+
+require "bundler/setup"
+require "libis/rosetta_checker"
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+
+# (If you use this, don't forget to add pry to your Gemfile!)
+# require "pry"
+# Pry.start
+
+require "irb"
+IRB.start(__FILE__)

data/bin/setup

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+set -euo pipefail
+IFS=$'\n\t'
+set -vx
+
+bundle install
+
+# Do any other automated setup that you need to do here

data/exe/rosetta_checker

@@ -0,0 +1,6 @@
+#!/usr/bin/env ruby
+
+require "bundler/setup"
+require "libis/rosetta_checker"
+
+Libis::RosettaChecker.run

data/lib/libis/rosetta_checker/files_to_ingest_cleanup.rb

@@ -0,0 +1,277 @@
+require 'optionparser'
+require 'digest'
+require 'bzip2/ffi'
+require 'zip'
+require 'oci8'
+require 'logging'
+require 'pathname'
+require 'csv'
+
+require_relative 'sub_command'
+require_relative 'options/files_to_ingest_cleanup'
+
+module Libis
+  module RosettaChecker
+    class FilesToIngestCleanup < SubCommand
+
+      def self.short_desc
+        'Report on files that are/are not ingested'.freeze
+      end
+
+      def self.command
+        'files2ingest'.freeze
+      end
+
+      def self.options_class
+        FilesToIngestCleanupOptions
+      end
+
+      def self.run
+        super do |cfg|
+          self.new(cfg).run(ARGV)
+        end
+      end
+
+      attr_accessor :cfg, :logger, :connection, :cursor, :report
+
+      def initialize(cfg)
+        @cfg = cfg
+
+        setup_logging
+
+        setup_db
+      end
+
+      def finalize
+        cursor.close if cursor
+        connection.logoff if connection
+      end
+
+      def run(argv)
+        raise ArgumentError, 'Need to specify at least a directory/file to parse' unless argv.size > 0
+        while (dir = argv.shift)
+          process_dir dir
+          next if argv.empty?
+          self.class.parse_options(argv)
+          setup_logging
+        end
+      end
+
+      protected
+
+      SQL_DATA = %w'ie_id rep_id fl_id original_name owner label group_id entity_type user_c'
+      CSV_HEADER = %w'parent_type parent file size md5 found name_match' + SQL_DATA
+
+      LOG_PATTERN = "[%d #%p] %-5l : %m\n".freeze
+
+      MSG_CALC_FC = '  - Calculating filesize and checksum'.freeze
+      MSG_CHCK_DB = '  - Checking file in DB'.freeze
+      MSG_DEFLATE = '  - Deflating'.freeze
+
+      FIND_SQL = <<-SQL
+        SELECT
+          CONCAT(sp.VALUE, ps.INDEX_LOCATION) as path,
+          ps.FILE_SIZE as filesize,
+          ps.CHECK_SUM_TYPE as checksum_type,
+          ps.CHECK_SUM as checksum,
+          sr.FILEORIGINALNAME as original_name,
+          ps.STORED_ENTITY_ID as fl_id,
+          cr.PID as rep_id,
+          ci.PID as ie_id,
+          ci.OWNER as owner,
+          ci.LABEL as label,
+          cf.GROUPID as group_id,
+        	ci.ENTITYTYPE as entity_type,
+        	ci.PARTITIONC as user_c
+        FROM
+        	V2KU_PER00.PERMANENT_INDEX ps
+        	LEFT JOIN V2KU_SHR00.STORAGE_PARAMETER sp ON sp.STORAGE_ID = ps.STORAGE_ID
+        	LEFT JOIN V2KU_REP00.HDESTREAMREF sr ON sr.PID = ps.STORED_ENTITY_ID
+        	LEFT JOIN V2KU_REP00.HDECONTROL cf ON cf.PID = ps.STORED_ENTITY_ID
+        	LEFT JOIN V2KU_REP00.HDECONTROL cr ON cr.PID = cf.PARENTID
+        	LEFT JOIN V2KU_REP00.HDECONTROL ci ON ci.PID = cr.PARENTID
+        WHERE
+        	sp."KEY" = 'DIR_ROOT'
+        AND	ps.FILE_SIZE = :filesize
+        AND ps.CHECK_SUM = :checksum
+        AND ps.CHECK_SUM_TYPE = 'MD5'
+        AND cf.OBJECTTYPE = 'FILE'
+        AND cr.OBJECTTYPE = 'REPRESENTATION'
+        AND ci.OBJECTTYPE = 'INTELLECTUAL_ENTITY'
+      SQL
+
+      def setup_logging
+        Logging.logger.root.level = :info
+        @logger = Logging.logger[self.class.command]
+        @logger.appenders = [Logging.appenders.stdout]
+        Logging.appenders.stdout.level = (@cfg.quiet ? :warn : :info)
+        @cfg.log_file = nil if @cfg.log_file&.chomp&.strip&.empty?
+        @logger.add_appenders Logging.appenders.file(
+            @cfg.log_file,
+            truncate: false,
+            layout: Logging.layouts.pattern(pattern: LOG_PATTERN)
+        ) if @cfg.log_file
+      end
+
+      def setup_db
+        @connection = OCI8.new(@cfg.dbuser, @cfg.dbpass, @cfg.dburl)
+        @cursor = @connection.parse(FIND_SQL)
+        @cursor.prefetch_rows = 10
+      end
+
+      def process_dir(dir)
+        if File.directory?(dir)
+          unless Dir.exist?(dir)
+            logger.error "Directory '#{dir}' does not exist"
+            return nil
+          end
+          unless File.readable?(dir)
+            logger.error "Directory '#{dir}' cannot be read"
+            return nil
+          end
+          logger.info "Processing dir '#{dir}'"
+          Dir.entries(dir).each do |entry|
+            next if %w'. ..'.include? entry
+            path = File.join dir, entry
+            begin
+              process_dir path if @cfg.recursive
+              next
+            end if File.directory?(path)
+            process_file path
+          end
+        elsif dir[0] == '@'
+          file = to_file(dir[1..-1])
+          return nil unless file
+          logger.info "Processing input file '#{file}'"
+          File.open(file, 'r').each_line do |line|
+            process_file(line.chomp, File.dirname(file))
+          end
+        elsif File.file?(dir)
+          process_file(dir)
+        else
+          raise ArgumentError, "Argument '#{dir}' should refer to an existing and readable file or directory"
+        end
+      end
+
+      def to_file(file, *search_dirs)
+        if File.exist?(file)
+          return file if File.readable?(file)
+          logger.error "File '#{file}' cannot be read"
+          return nil
+        end
+        search_dirs.each do |dir|
+          f = File.join(dir, file)
+          if File.exist? f
+            return f if File.readable?(f)
+            logger.error "File '#{f}' cannot be read"
+            return nil
+          end
+        end
+        logger.error "File '#{file}' does not exist"
+        nil
+      end
+
+      def process_file(_file, *search_dir)
+        file = to_file(_file, *search_dir)
+        return unless file
+
+        if File.directory?(file)
+          process_dir(file)
+          return
+        end
+
+        info = {
+            parent_type: 'D',
+            parent: File.dirname(file),
+            file: File.basename(file)
+        }
+
+        logger.info "- #{file}"
+        if File.extname(file) == '.bz2'
+          logger.info MSG_DEFLATE
+          info[:size] = 0
+          reader = Bzip2::FFI::Reader.open file
+          md5 = Digest::MD5.new
+          logger.info MSG_CALC_FC
+          while (data = reader.read 2048000) do
+            info[:size] += data.length
+            md5 << data
+          end
+          reader.close
+          info[:parent_type] = 'F'
+          info[:parent] = file
+          info[:file] = File.basename file, '.bz2'
+          info[:md5] = md5.hexdigest
+          check_file info
+        elsif File.extname(file) == '.zip'
+          logger.info '  - Unpacking'.freeze
+          info[:parent_type] = 'Z'
+          info[:parent] = file
+          Zip::File.open(file) do |zip|
+            zip.each do |entry|
+              next if entry.directory?
+              info[:file] = entry.name
+              logger.info "- #{file}/#{entry.name}"
+              logger.info MSG_CALC_FC
+              info[:size] = 0
+              md5 = Digest::MD5.new
+              reader = entry.get_input_stream
+              while (data = reader.read 2048000) do
+                info[:size] += data.length
+                md5 << data
+              end
+              reader.close
+              info[:md5] = md5.hexdigest
+              check_file info
+            end
+          end
+        else
+          begin
+            logger.info MSG_CALC_FC
+            info[:size] = File.size file
+            info[:md5] = Digest::MD5.file file
+            check_file info
+          rescue Exception
+            logger.error "Could not access file '#{file}'"
+          end
+        end
+      end
+
+      def check_file(info)
+        logger.info MSG_CHCK_DB
+
+        cursor.bind_param(':filesize', info[:size].to_i)
+        cursor.bind_param(':checksum', info[:md5].to_s)
+
+        cursor.exec
+
+        while (found = cursor.fetch_hash)
+          SQL_DATA.each {|x| info[x.to_sym] = found[x.upcase]}
+          logger.info "    found match: #{info[:ie_id]}/#{info[:rep_id]}/#{info[:fl_id]}"
+          if info[:original_name] =~ Regexp.new(info[:file].split(/[ #._-]/).join('.*'))
+            logger.info "    name matches: #{info[:original_name]}"
+            info[:name_match] = true
+          else
+            info[:name_match] = false
+          end
+
+        end
+
+        info[:found] = cursor.row_count
+
+        to_report(info)
+
+      end
+
+      def to_report(info = nil)
+        return unless @cfg.report
+        unless @report
+          @report ||= CSV.open(@cfg.report_file, 'wb')
+          @report << CSV_HEADER
+        end
+        @report << CSV_HEADER.map {|x| info[x.to_sym]} if info
+      end
+
+    end
+  end
+end